kesahan & kebolehpercayaan pembentukan instrumen, kesahan dan kebolehpercayaan instrumen

Powerpoint Templates Mengilham harapan, mencipta masa depanInspiring Futrures, Inspiring Possibilities

Instrumentasi: Pembentukan Instrumen, Kesahan dan Kebolehpercayaan Instrumen

http://www.powerpointstyles.com/

Powerpoint Templates

Main Steps in Quantitative Research:

1. Theory2. Hypothesis3. Research design4. Plan/Develop/Create measures of concepts5. Select research site(s)6. Select research subjects/respondents7. Administer research instruments/ collect data8. Process data9. Analyse data10. Write up findings and conclusions



Examples of Quantitative Research Methods:

ExperimentsSocial surveys

Cross-sectionalComparative (cross-national)Longitudinal

Content AnalysisStatistical AnalysisOfficial Statistics

DemographyEpidemiology

Field stimulations Structured Interviews and Observation



Terms “survey” and “questionnaire” are often used interchangeably

“Survey” refers to a descriptive research technique or methodology (also includes interviews, and focus groups)“Questionnaire” refers to a paper and pencil instrument through which information is obtained

“Survey”/Tinjauan



• Useful for surveying groups of any size

• Typically designed to determine opinions, attitudes or present practices

• Limitation: rely on self report information v. observed behavior

• Challenges to validity become a concern



Validity: the extent to which the test measures what it significances/meaning/senses to measureInternal Validity: Relates to instrument (questionnaire) quality. Are the appropriate questions asked clearly and logically?External validity: can the results be generalized?

Measurement error: difference between the characteristics of your sample and the characteristics of the population

Non-response bias: answers of respondents do not mirror the characteristics of those who refused participation.

Validity



Create a questionnaire that is clear and logical

Content Relevance And Content Representativeness

Content Relevance: are the questions relevant to the purpose of the questionnaire? Unimportant content should be eliminated.

Content Representativeness: Are the questions an adequate representation of the universe of possible questions?Solicit the input of experts in scale development and the content of the questionnaire for an objective evaluation



• Determine the Objective– Is the development of a new questionnaire

• Necessary?

• Feasible ? (practicable/viable/achievable/possible..etc)

– What do you want to ask and of whom?– Consider how the data will be analyzed. What

information is needed?

• Select mode of administration– Web Based?– Paper and pencil?– Combination?

QUESTIONNAIRE PLANNING



QUESTIONNAIRE PLANNING

• Determine the Sampling Methodology– To whom will you send the questionnaire? Who can supply

the information?

– Will everyone be surveyed?• ALL current students? ALL alumni?

• If yes, this is a census of the finite population

– The selection of a representative sample of the population may be more effective based on time and cost of administration

• Decision depends on the size of the finite population



CONSTRUCTING the QUESTIONNAIRE

• Consider what objective each question measures• Three to five questions per objective is recommended

• How many questions?– Find a balance: long enough to obtain the necessary

information but short enough so respondents won’t lose interest.

– Questionnaires that are too long won’t be completed

– Goal: 25-30 minutes MAX



DESIGNING THE QUESTIONNAIRE

• Question Formats: Open Ended Questions– Allow respondents to express feelings and expand on ideas. – Question should be phrased to avoid a one word response.

• “What aspects of the Sociology program best prepared you for your current position?”

– Open ended questions at the beginning of the questionnaire may be discouraging to respondent – include one or two at the end

– Analysis of responses can be challenging



QUESTIONNAIRE DEVELOPMENT

• Question Formats: Closed Questions– Ranking

• Forces the respondent to rank order responses based on a value judgment.

• Rank the following items with regard to how you spend time at Bucknell from 1 (spend the most time) to 5 (spend the least time).

_____ Studying_____ Socializing_____ Volunteering_____ Working at a part time job_____ Exercising




• Question Formats: Closed Questions– Checklist

• A number of possible answers are provided and the respondent is asked to choose one or choose all that apply.

• How did you finance your Bucknell University education? Choose all that apply.

_____ Parental contribution_____ Student Loan_____ Work Study_____ Off campus employment_____ Other (please specify): _____________




• Question Formats: Closed Questions– Scaled items

• Very common

• Indicate strength of agreement or disagreement with a statement.

• Numbers are assigned to each response on the continuum

• The English curriculum at Bucknell University adequately prepared me for my current position. 1 2 3 4 5

Strongly

Disagree Disagree

Neutral / No

Opinion AgreeStrongly

Agree




• Question Formats: Closed Questions– Suggestions for writing scaled items

• Use positively worded or negatively worded items consistently

• Use an odd number of responses along the continuum to allow for a neutral response

• Anchors may be different based on focus of the item (Strongly Agree – Strongly Disagree; Frequently – Never; Very Good – Very Poor)

• Keep items short




• Question Formats: Categorical– Typical demographic questions:

• Yes/No/?, Male/Female, F/S/J/S/G– Answers fall into certain categories – respondents pick the one

that applies to them– Analyzed using frequency, proportion, percentages– 2 considerations:

• Questions on a continuum should stay on a continuum if they will be directly analyzed

– Example: Age – allow respondent to write in their age rather than check off a box for their age range

• Categories must be collectively exhaustive & mutually exclusive (next slide)




• Problematic:

• Number of hours spent studying per week:– 18 – 21 – 21 – 24– 24 – 27

• Categories are not mutually exclusive

• Corrected:

• Number of hours spent studying per week:– 18 – 21– 22 – 25– 26 – 29



IMPORTANT CONSIDERATIONS

• APPEARANCE AND DESIGN– Have clear directions that are age appropriate & easy to

follow– Format, size, reproduction should facilitate completion

• No less than 12 pt• Times, Arial, Courier

• PILOT STUDY– Give the questionnaire to a few people to try it out– Have them evaluate readability and clarity of questions– Estimate time required to complete the questionnaire– Suggestions for revisions



SENDING IT OUT

• INITIAL CONTACT / COVER LETTER– Short & to the point– Identify person by name– Convince respondent that participation is

important– Ensure confidentiality– Deadline for return of questionnaire– SASE for return (mailed questionnaires only)



FINAL THOUGHTS

• FOLLOW UP PROCEDURES– Send to all participants to preserve confidentiality (“If you

haven’t already returned the questionnaire…..”)

– May include a second copy if available– Usually done in 2 week intervals

• RESPONSE RATE FOR MAILED QUESTIONNAIRES– Typically low - 30% is considered good

– Higher response rate with web-based questionnaires– Good follow-up increases response rate



SUMMARY OF STEPS

1. Determine the Objective (consider analyses)

2. Determine the Mode of Administration

3. Determine the Sampling Methodology

4. Construct the Questionnaire

5. Institutional Approval

6. Conduct the Pilot Study

7. Write the Initial Communication

8. Send the Questionnaire

9. Follow up

10. Analyze the Results



Research design and methods

Develop your own research design and methods of your study



DEVELOPING QUALITATIVE INSTRUMENT



Main Steps in Qualitative Research:

1. General research question

2. Select relevant site(s) and subjects

3. Collection of relevant data

4. Interpretation of data

5. Conceptual and theoretical work

6. Tighter specification of the research question

7. Collection of further data

8. Conceptual and theoretical work

9. Write up findings



Examples of Qualitative Research

In-depth InterviewsFocus GroupsEthnography/Field ResearchHistorical-Comparative ResearchDiscourse AnalysisNarrative AnalysisMedia Analysis



RELIABILITY & VALIDITY OF INSTRUMENT - QUANTITATIVE

INSTRUMENT



Criteria of Measurement QualityCriteria of Measurement Quality

• How do we judge the relative success (or How do we judge the relative success (or failure) in measuring various concepts?failure) in measuring various concepts?– Reliability – consistency of measurement Reliability – consistency of measurement – Validity – confidence in measures and designValidity – confidence in measures and design



Reliability and ValidityReliability and Validity

• Reliability focuses on measurementReliability focuses on measurement• Validity also extends to:Validity also extends to:

• Precision in the design of the study – ability to Precision in the design of the study – ability to isolate causal agents while controlling other isolate causal agents while controlling other factorsfactors

– (Internal Validity)(Internal Validity)

• Ability to generalized from the unique and Ability to generalized from the unique and idiosyncratic settings, procedures and idiosyncratic settings, procedures and participants to other populations and conditionsparticipants to other populations and conditions

– (External Validity)(External Validity)



ReliabilityReliability

• Consistency of MeasurementConsistency of Measurement– Reproducibility over timeReproducibility over time– Consistency between different Consistency between different

coders/observerscoders/observers– Consistency among multiple indicatorsConsistency among multiple indicators

• Estimates of ReliabilityEstimates of Reliability– Statistical coefficients that tell use how Statistical coefficients that tell use how

consistently we measured somethingconsistently we measured something



Measurement ValidityMeasurement Validity

• Are we really measuring concept we defined?Are we really measuring concept we defined?– Is it a valid way to measure the concept?Is it a valid way to measure the concept?

• Many different approaches to validationMany different approaches to validation– Judgmental as well as empirical aspectsJudgmental as well as empirical aspects



Key to Reliability and ValidityKey to Reliability and Validity

• Concept explicationConcept explication– Thorough meaning analysisThorough meaning analysis

• Conceptual definition:Conceptual definition:– Defining what a concept means Defining what a concept means

• Operational definition: Operational definition: – Spelling out how we are going to measure Spelling out how we are going to measure

conceptconcept



Four Aspects of Reliability:Four Aspects of Reliability:

• 1. Stability1. Stability

• 2. Reproducibility2. Reproducibility

• 3. Homogeneity3. Homogeneity

• 4. Accuracy4. Accuracy



1. Stability1. Stability

• Consistency across timeConsistency across time– repeating a measure at a later time to repeating a measure at a later time to

examine the consistency examine the consistency – Compare time 1 and time 2Compare time 1 and time 2



2. Reproducibility2. Reproducibility

• Consistency between observersConsistency between observers

• Equivalent application of measuring Equivalent application of measuring devicedevice– Do observers reach the same conclusion?Do observers reach the same conclusion?– If we don’t get the same results, what are we If we don’t get the same results, what are we

measuring?measuring?• Lack of reliability can compromise validityLack of reliability can compromise validity



3. Homogeneity3. Homogeneity

• Consistency between different measures Consistency between different measures of the same conceptof the same concept– Different items used to tap a given concept Different items used to tap a given concept

show similar results – ex. open-ended and show similar results – ex. open-ended and closed-ended questionsclosed-ended questions



4. Accuracy4. Accuracy

• Lack of mistakes in measurementLack of mistakes in measurement

• Increased by clear, defined proceduresIncreased by clear, defined procedures– Reduce complications that lead to errorsReduce complications that lead to errors

• Observers must have sufficient:Observers must have sufficient:– TrainingTraining– MotivationMotivation– ConcentrationConcentration



Increasing ReliabilityIncreasing Reliability

• General:General:– Training coders/interviewers/lab personnelTraining coders/interviewers/lab personnel– More careful concept explication More careful concept explication

(definitions)(definitions)– Specification of procedures/rulesSpecification of procedures/rules– Reduce subjectivity (room for Reduce subjectivity (room for

interpretation)interpretation)• Survey measurement:Survey measurement:

– Increase the number of items in scaleIncrease the number of items in scale– Weeding out bad items from “item pool”Weeding out bad items from “item pool”

• Content analysis coding:Content analysis coding:– Improve definition of content categoriesImprove definition of content categories– Eliminate bad codersEliminate bad coders



Indicators of ReliabilityIndicators of Reliability

• Test-retestTest-retest– Make measurements more than once and see if they Make measurements more than once and see if they

yield the same resultyield the same result

• Split-halfSplit-half– If you have multiple measures of a concept, split If you have multiple measures of a concept, split

items into two scales, which should then be correlateditems into two scales, which should then be correlated

• Cronbach’s Alpha or Mean Item-total CorrelationCronbach’s Alpha or Mean Item-total Correlation



Reliability and ValidityReliability and Validity

• Reliability is a necessary condition for Reliability is a necessary condition for validityvalidity– If it is not reliable it cannot be validIf it is not reliable it cannot be valid

• Reliability is NOT a sufficient condition for Reliability is NOT a sufficient condition for validityvalidity– If it is reliable it may not necessarily be validIf it is reliable it may not necessarily be valid

• Example:Example:– Bathroom scale, old springs Bathroom scale, old springs



Not Reliable or ValidNot Reliable or Valid



Reliable but not ValidReliable but not Valid



Reliable and ValidReliable and Valid



Types of ValidityTypes of Validity

• 1. Face validity1. Face validity• 2. Content validity2. Content validity• 3. Pragmatic (criterion) validity3. Pragmatic (criterion) validity

– A. Concurrent validityA. Concurrent validity– B. Predictive validityB. Predictive validity

• 4. Construct validity4. Construct validity– A. Testing of hypothesesA. Testing of hypotheses– B. Convergent validityB. Convergent validity– C. Discriminant validityC. Discriminant validity



Face ValidityFace Validity

• Subjective judgment of experts about:Subjective judgment of experts about:– ““what’s there”what’s there”– Do the measures make sense?Do the measures make sense?

• Compare each item to conceptual definitionCompare each item to conceptual definition– Do it represent the concept in question?Do it represent the concept in question?

• If not, it should be droppedIf not, it should be dropped– Is the measure valid “on its face”Is the measure valid “on its face”



Content ValidityContent Validity• Subjective judgment of experts about:Subjective judgment of experts about:

– ““what is not there”what is not there”• Start with conceptual definition of each Start with conceptual definition of each

dimension:dimension:– Is it represented by indicators at the Is it represented by indicators at the

operational level?operational level?– Are some over or underrepresented?Are some over or underrepresented?

• If current indicators are insufficient:If current indicators are insufficient:– develop and add more indicatorsdevelop and add more indicators

• Example--Civic Participation questions:Example--Civic Participation questions:– Did you vote in the last election?Did you vote in the last election?– Do you belong to any civic groups?Do you belong to any civic groups?– Have you ever attended a city council Have you ever attended a city council

meeting?meeting?– What about “protest participation” or “online What about “protest participation” or “online

organizing”?organizing”?



Pragmatic ValidityPragmatic Validity

• Empirical evidence used to test validityEmpirical evidence used to test validity– Compare measure to other indicatorsCompare measure to other indicators

• 1. Concurrent validity1. Concurrent validity– Does a measure predict simultaneous Does a measure predict simultaneous

criterion?criterion?• Validating new measure by comparing to existing measureValidating new measure by comparing to existing measure• E.g., Does new intelligence test correlate with established E.g., Does new intelligence test correlate with established

testtest

• 2. Predictive validity2. Predictive validity– Does a measure predict future criterion?Does a measure predict future criterion?

• E.g., SAT scores: Do they predict college GPA?E.g., SAT scores: Do they predict college GPA?



Construct ValidityConstruct Validity

• Encompasses other elements of validityEncompasses other elements of validity• Do measurements:Do measurements:

– A. Represent all dimensions of the conceptA. Represent all dimensions of the concept– B. Distinguish concept from other similar B. Distinguish concept from other similar

concepts concepts • Tied to meaning analysis of the conceptTied to meaning analysis of the concept

– Specifies the dimensions and indicators to be Specifies the dimensions and indicators to be testedtested

• Assessing construct validityAssessing construct validity– A. Testing hypothesesA. Testing hypotheses– B. Convergent validityB. Convergent validity– C. Discriminant validityC. Discriminant validity



A. Testing HypothesesA. Testing Hypotheses

• When measurements are put into practice:When measurements are put into practice:– Are hypotheses that are theoretically Are hypotheses that are theoretically

derived, supported by observations?derived, supported by observations?• If not, there is a problem with:If not, there is a problem with:

– A. TheoryA. Theory– B. Research design (internal validity)B. Research design (internal validity)– C. Measurement (construct validity?)C. Measurement (construct validity?)

• In seeking to examine construct validity:In seeking to examine construct validity:– Examine theoretical linkages of the Examine theoretical linkages of the

concept to othersconcept to others• Must identify antecedent and consequencesMust identify antecedent and consequences

– What leads to the concept?What leads to the concept?– What are the effects of the concept?What are the effects of the concept?



B. Convergent ValidityB. Convergent Validity

• Measuring a concept with different methodsMeasuring a concept with different methods– If different methods yield the same results:If different methods yield the same results:

• than convergent validity is supportedthan convergent validity is supported– E.g., Survey items measuring Participation:E.g., Survey items measuring Participation:

• VotingVoting

• Donating to money to candidatesDonating to money to candidates• Signing petitionsSigning petitions• Writing letters to the editorWriting letters to the editor• Civic group membershipsCivic group memberships

• Volunteer activitiesVolunteer activities



C. Discriminant (Divergent) ValidityC. Discriminant (Divergent) Validity

• Measuring a concept to discriminate that Measuring a concept to discriminate that concept from other closely related conceptsconcept from other closely related concepts– E.g., Measuring Maternalism and Paternalism E.g., Measuring Maternalism and Paternalism

as distinct conceptsas distinct concepts



Dimensions of Validity for Dimensions of Validity for Research DesignResearch Design

• InternalInternal– Validity of research designValidity of research design

• Validity of sampling, measurement, Validity of sampling, measurement, proceduresprocedures

• ExternalExternal– Given the research design, how valid areGiven the research design, how valid are

• Inferences made from the conclusionsInferences made from the conclusions• Implications for real worldImplications for real world



Internal and External Validity in Internal and External Validity in Experimental DesignExperimental Design

• Internal validity:Internal validity:

– Did the experimental treatment make a Did the experimental treatment make a difference?difference?

• Or is there an internal design flaw that invalidates the results?Or is there an internal design flaw that invalidates the results?

• External validity:External validity:

– Are the results generalizable?Are the results generalizable?• Generalizable to:Generalizable to:

– What populations?What populations?

– What situations?What situations?

• Without internal validity, there is no external validityWithout internal validity, there is no external validity



RELIABILITY & VALIDITY OF

INSTRUMENT - QUALITATIVE INSTRUMENT



Core Qualitative Methods

• Semi- or Un-structured, Open-Ended:

– In-depth Interviews (in the field, face-to-face)

– Participant Observation (field/site visits)

– Archival Research (document review and analysis)



Guiding Principles

• Qualitative research designs consider ways to foster:– Reflexivity (an ongoing process of reflecting on

the researcher’s subjective experience, ways to broaden and enhance this source of knowing, & examining how it informs research)

– Iteration (a spiraling process: sequential and repetitive steps in examining preliminary findings for the purposes of guiding additional data collection and analysis)

– Intersubjectivity (a process of reaching a shared/ “objective” agreement about how to assign meaning to a social experience - with insiders and outsiders)



The Iterative Process of Qualitative Research: A Model

Data CollectionData Collection

ReflectionReflection

AnalysisAnalysis



Qualitative Research Techniques

• Instrumentation:

– Key Informants (question development and piloting of instrument)

– Unstructured to Semi-structured

– Probing

– Data Processing and Analytic Tools



Qualitative Research Techniques/ Considerations

• Sampling

– Single v. Multiple Cases (not an individual) – Expert and Key Informants (identification and

recruitment of sample)

– Roles of the Researcher (identification and recruitment of sample)




• Data Collection

– Participants as Data Collectors – Field Notes (personal reflections, observations,

emerging concepts/theories)

– Debriefing (a participant, a participating researcher, a non-participating researcher)




• Analysis

– Key Informant Feedback

– Codebooks (specifies definitions and relationships of concepts and terms)

– Memos (emerging patterns, concepts; documentation of analytic pathways)

– Case Analysis Meeting (a meeting of a research team for the purposes of reflecting on analytic process, tools, and findings)

– Matrices or Diagrams (to identify and examine time sequencing, the structure of relationships, conditions of cross case events)



Three Qualitative Methods: Ethnography

• Focus: study human behaviour in the cultural context in which it is embedded

• Ethnography is the work of describing a culture – the way of life of a cultural group– Associated with Cultural Anthropology

• Example: David Counts and Dorothy Ayer Counts: “An Ethnography of RVing Seniors”



• Focus: develop a theory to explain underlying social processes of a cultural group

• Useful in areas where little is known or when a new perspective is needed– Used for exploratory, descriptive studies

• Because the theory emerges from the data, it is said to be grounded in the data– Foundation in Symbolic Interactionism

• Example: Kerry Daly and the social construction of fatherhood

Three Qualitative Methods: Grounded Theory



Three Qualitative Methods: Phenomenology

• Focus: reveal the meaning of the lived experience from the perspective of participants

• Describe the essences of lived experience

– Essences: elements related to the true meaning of something that gives common understanding to the phenomenon under study

– Conveyed with descriptive language

– Drawn from Philosophy; used across disciplines• Example: J.E. Solchany: A phenomenological study of

women’s preadoptive experiences



Data Collection Methods in Qualitative Studies

• Three data collection strategies introduced:1. Participant observation

2. In-depth interviews

3. Focus group interviews

• Qualitative researchers may combine more than one method



Data Collection Methods:1. Participant Observation

• Intensive, usually long term, examination of a social group, an organization, etc.

• Researcher becomes a participant in the lives of group members– Observes their behaviour and learns meaning

systems (which are tied to language)

• Most closely associated with Ethnography, as developed in Classical Anthropology

• Now done in a variety of disciplines



1. Participant Observation (cont’d)

• Many classic participant observations studies – E.g., Asylums (Erving Goffman), Tally’s

Corner (Elliot Liebow), Street Corner Society (William F. Whyte), etc.

• Today most ethnographers take an overt role– I.e., their identity as a researcher is known

to the people being studied• Covert participation (i.e., identity concealed

from participants) is fraught with ethical issues – e.g., Humphrey’s “Tearoom Trade”



Steps Involved in Participant Observation Research

A. Gaining entry into the group

B. Developing and maintaining rapport

C. Developing a method for taking field notes

D. Integrating data collection and data analysis



Steps in Participant Observation:A. Gaining Entry into the Group

• Take into consideration the type of group– formal organizations require formal entry;

involves letter writing, permission requests, etc.

– Informal groups – different strategy needed• Access may be gained through a gatekeeper

(an individual with special status)• Want to involve key informants (those who are

most knowledgeable about the group)



Steps in Participant Observation: B. Developing/Maintaining Rapport

• Researcher must work hard to develop and maintain good relationships in the field– E.g., be sure not to become associated

with one faction in a group or organization

• Researcher could be blamed for problems that arise in the setting



Steps in Participant Observation: C. Strategies for Taking Field Notes

Field notes – integral to participant observation• Include descriptions and interpretations of individuals,

interactions, and events– Distinguish descriptions from interpretations

• Record time and location of observations, as well as key information (weather, events happening and their significance)

• Keep theoretical memos – which are the tentative interpretations emerging and being assessed through further data collection



Field Notes (cont’d)

• May not be possible or advisable to take notes while in the field– Important that they be done as soon after

field observation as possible

• Note-taking is time-consuming because it is integral to guiding the data collection and continuing the analysis– E.g., field notes for When Prophecy Failed

were well over 1,000 typed pages



Steps in Participant Observation: D. Integrating Data Collection and

Analysis• Organizing field notes into different types of files facilitates data analysis

• Master field file – complete journal of field notes; number pages and include entry dates

• Background, history file – subfile organizing background material

• Key character files – subfiles on key players in the group or organization

• Analytic files – subfiles for different types of observations or relationships



Data Collection Methods: 2. In-depth Interviews

• Some studies cannot employ the participant observation method– E.g., Desroches’s study of bank robbers

• In-depth interviews allow participants to describe their experiences and the meaning of events taking place in their lives– Verbatim quotes capture the language and

meaning expressed by participants• Interviews are flexible and allow for probing

– Interview method is quite diverse, adaptive



2. In-depth Interviews (cont’d)

• Spradley – three key elements for the interview method to be successful

1. Explicit purpose – researcher and informant are aware that the discussion has a purpose

2. Ethnographic explanations – researcher tries out explanations on the participants to see if they make sense– Encourage the informants to use colloquial

language, and teach the researcher its meaning



2. In-depth Interviews (cont’d)

3. Ethnographic questions include:

i. Descriptive questions – ask participants to describe their experiences (e.g., their ideas, circumstances, viewpoints, dilemmas, etc)

ii. Structural questions – ask participants how they organize their world (e.g., activities)

iii. Contrast questions – ask participants what is meant by specific terminology

• Prus & Grills stress the value of the interview method in a multi-method approach



Data Collection Methods:3. Focus Group Interviews

• Interview format, but in a group setting– 6-12 participants with common experience

• Dates back to the 1940s – used to assess effectiveness of morale-boosting radio shows– 1970s onward – used by market researchers– 1980s onward – used by academics

• Transcript of discussion is the data– Plus accompanying notes– Use content analysis or grounded theory

approach to analyze the data



3. Focus Group Interviews (cont’d)

• Strengths:– Open-ended question

• Spontaneously deal with issues as they arise– Cost-effective method of collecting data– Less time-consuming

• Weaknesses:– One or two participants may dominate– Not done in a natural setting, so little “observation” to

help understand the experience of the participants



Implications for Handling Threats to Validity and Reliability

• In quantitative research, threats to validity are addressed by prior design features (such as randomization and controls)



Implications for Handling Threats to Validity and Reliability

• In qualitative research, such prior elimination of threats to validity is less possible because:

– qualitative research is more inductive, and

– it focuses primarily on understanding particulars rather than generalizing to universals.

• Qualitative researchers view “threats” as an opportunity for learning

- e.g. researcher effects and bias are part of the story that is told; they are not “controlled for”



Validity

• In general, validity concerns the degree to which an account is accurate or truthful

• In qualitative research, validity concerns the degree to which a finding is judged to have been interpreted in a correct way



Assessing the Validity of Qualitative Research

Can another research read your field (and other types of) notes (i.e., the explication of your logic) and come to the same understandings of a given phenomenon?

• Concern about validity (as well as reliability) is the primary reason thick description is an essential component of the qualitative research enterprise

• Resources:

– Handout: Different Types of Notes

– Example: ACY Site Visit Toolkit



Major Types of Validity in Qualitative Research

• Descriptive Validity

• Interpretive Validity

• Theoretical Validity

• External Validity (i.e., generalizability)



Descriptive Validity

• Concerned with the factual accuracy of an account (that is, making sure one is not making up or distorting the things one hears and sees)

• All subsequent types of validity are dependent on the existence of this fundamental aspect of validity



Descriptive Validity

Behavior must be attended to, and with some exactness, because it is through the flow of behavior – or, more precisely, social action – that cultural forms find articulation.

Geertz 1973:17



Interpretive Validity

• Interpretive accounts are grounded in the language of the people studied and rely, as much as possible, on their own words and concepts

• At issue, then, is the accuracy of the concepts as applied to the perspective of the individuals included in the account



Interpretive Validity: Design Consideration

• While the relevant consensus about the terms used in description rests in the research community, the relevant consensus for the terms used in interpretation rests, to a substantial extent, in the community studied

• An important design element, for increasing interpretive validity, therefore, is to employee, at some level/to some degree, a participatory research approach (e.g., through member checks, peer to peer research model, etc.)



Theoretical Validity

• Theoretical understanding goes beyond concrete description and interpretation; it’s value is derived based on its ability to explain succinctly the most amount of data

• A theory articulates/formulates a model of relationships as they are postulated to exist between salient variables or concepts

• Theoretical validity is thus concerned, not only with the validity of the concepts, but also their postulated relationships to one another, and thus its “goodness of fit” as an explanation



Major Threats to Validity

• Type I error: believing a principle to be true when it is not (i.e., mistakenly rejecting the null hypothesis)

• Type II error: rejecting a principle when, in fact, it is true

• Type III error: asking the wrong question



Triangulation: An Important Theoretical Validity Check

• Case example: Parable of the blind men and the elephant



Triangulation: An Important Theoretical Validity Check

The most fertile search for validity comes from a combined series of different measures, each with its own idiosyncratic weaknesses, each pointed to a single hypothesis. When a hypothesis can survive the confrontation of a series of complementary methods of testing, it contains a degree of validity unattainable by one tested within the more constricted framework of a single method.

Webb et al. 1966:174



External Validity in Qualitative Research

• There is broad agreement that generalizability (in the sense of producing laws that apply universally) is not a useful standard or goal for qualitative research

• This is not to say, however, that studies conducted to examine a particular phenomenon in a unique setting cannot contribute to the development of a body of knowledge accumulating about that particular phenomenon of interest

• Consensus appears to be emerging that for qualitative researchers generalizability is best thought of as a matter of the “fit” between the situation studied and others to which one might be interested in applying the concepts and conclusions of that study.



Enhancing External Validity

• Thick descriptions are crucial.

• Such descriptions of both the site in which the studies are conducted and of the site to which one wishes to generalize (or apply one’s findings) are critical in allowing one to search for the similarities and differences between the situations.

• Analysis of these similarities and differences makes it possible to make a reasoned judgment about the extent to which we can use the findings from one study as a working hypothesis about what might occur in another situation.



Multi-site Studies: Another Way to Enhance Generalizability

• A finding emerging repeatedly in the study of numerous sites would appear to be more likely to be a good working hypothesis about some as yet unstudied site than a finding emerging from just one or two sites.

• A finding emerging from the study of several very heterogeneous sites would be more robust and, thus, more likely to be useful in understanding various other sites than one emerging from the study of several very similar sites.

• Heterogeneity may be obtained by creating a sampling frame that maximizes the variation inherent in the sample, specifically in terms of potentially theoretically important dimensions



Reliability

• Reliability concerns the ability of different researchers to make the same observations of a given phenomenon if and when the observation is conducted using the same method(s) and procedure(s)



Enhancing the Reliability of Qualitative Research

• Researchers can enhance the reliability of their qualitative research by:

– Standardizing data collection techniques and protocols

– Again, documenting, documenting, documenting (e.g., time day and place observations made)

– Inter-rater reliability (a consideration during the analysis phase of the research process)


kesahan & kebolehpercayaan pembentukan instrumen, kesahan dan kebolehpercayaan instrumen

Education