How to Design a Survey Instrument That Passes Your Panel’s Scrutiny

At StatAce, we see the same pattern almost every term. A student walks into their proposal defense with a questionnaire that took weeks to build, only to have the panel spend twenty minutes picking it apart: unclear items, no expert validation, a reliability plan that doesn’t exist yet, a scale borrowed from a foreign study with no adaptation for the Filipino context. None of these are hard problems to avoid. They’re just steps that get skipped when students are racing to finish their proposal.

This article walks through the instrument-building process in the order a panel actually expects to see it, from defining your constructs to piloting your final version.

Step 1: Define Your Constructs Before You Write a Single Item

The most common root cause of a weak instrument isn’t bad item wording; it’s starting to write items before the underlying construct is clearly defined. Before drafting anything, write out, in your own words, a precise definition of each variable you intend to measure. If your study measures “employee engagement,” can you state exactly what that means in your study, and how it differs from related constructs like job satisfaction or organizational commitment? Panels will ask this, and “I used a definition from a journal article” is a weaker answer than being able to explain the definition yourself.

Once your constructs are clearly defined, break each one into its dimensions, if it has more than one. A construct like “service quality,” for instance, is often multidimensional (reliability, responsiveness, assurance, tangibles, empathy, if you’re using the well-known SERVQUAL framework). Each dimension typically needs its own cluster of items.

Step 2: Decide Between Adopting, Adapting, or Developing Your Instrument

You generally have three options, and your panel will want to know which one you chose and why.

Adopting an existing, validated instrument. This is often the safest route for a graduate thesis, provided the instrument’s original context is reasonably close to yours, and you have permission to use it (many published scales require author permission or licensing, so check this early). Even with an adopted scale, panels expect you to re-establish reliability with your own sample rather than simply citing the original study’s Cronbach’s alpha.

Adapting an existing instrument. This means modifying an established scale to fit your context, translating it into Filipino or a local dialect, changing industry-specific terms, or adjusting the number of items. Adaptation requires more validation work than straight adoption: back-translation if you’re changing languages, and expert review of the adapted items specifically.

Developing a new instrument. This is the most rigorous path and the one panels scrutinize most closely. If no existing instrument adequately captures your construct, developing your own is legitimate, but it requires a full validation process: item generation from theory and literature, expert content review, pilot testing, and reliability analysis, at minimum. Budget significant time for this option specifically because of the extra validation steps involved.

Step 3: Draft Your Items

With your constructs and approach decided, draft your items using these principles:

• One idea per item. Avoid double-barreled items that ask about two things at once (see our companion article on Likert scale mistakes for a deeper look at this).
• Plain, direct language. Avoid jargon your respondents may not share, even if it’s standard terminology in your field.
• Neutral phrasing. Avoid leading or loaded wording that nudges respondents toward a particular answer.
• Balanced positive and negative wording, if you’re using this technique to reduce straight-lining, with a clear plan for reverse-scoring built in from the start.
• Enough items per construct. A single item per construct generally cannot produce a meaningful reliability estimate. Three to five items per dimension is a common minimum for social science research.

Step 4: Establish Content Validity Through Expert Review

This is the step students skip most often, and it’s usually the first thing a thorough panel asks about.

Identify three to five experts with relevant knowledge; this could be content experts in your field, methodologists, or both, and have them rate each item for relevance and clarity, typically on a 3-point or 4-point scale. Summarize their ratings using a content validity index (CVI), either per item (I-CVI) or for the scale overall (S-CVI). Revise or remove items that experts flag as unclear, redundant, or irrelevant.

Document this process. Keep a record of who your experts were (their credentials, not necessarily their names if confidentiality matters), what changes you made based on their feedback, and your final CVI values. This becomes an appendix item your panel will expect to see, and it’s also one of the easiest points to score well on since it doesn’t depend on your data at all.

Step 5: Translate and Back-Translate, If Applicable

If your instrument will be administered in Filipino, Bisaya, or another local language, and especially if you’re adapting a foreign-language instrument, use a back-translation process: translate the instrument into the target language, then have a separate, independent translator translate it back into the original language without seeing the original wording. Compare the two versions and resolve any discrepancies. This is standard practice in cross-cultural research and something panels in the Philippines are increasingly likely to ask about directly, given how much research here involves bilingual or multilingual respondents.

Step 6: Pilot Test Before Full Data Collection

Pilot testing is not optional if you want a defensible instrument, and it should happen with a separate sample from your actual study, typically 20 to 30 respondents who share key characteristics with your target population but won’t be included in your final dataset.

During the pilot, check for:

• Clarity issues. Did respondents ask clarifying questions, skip items, or seem confused? Consider a short debrief with a few pilot respondents asking what they thought each item meant.
• Completion time. Is the survey a reasonable length? Excessive length increases fatigue-driven straight-lining and dropout.
• Reliability. Compute Cronbach’s alpha for each subscale using the pilot data. A commonly used threshold is 0.70 or higher, though this should be interpreted alongside the number of items and the nature of the construct.
• Preliminary factor structure, if your sample size and study design allow for it at this stage, to check whether items are loading onto the dimensions you expected.

Items that perform poorly in the pilot, low item-total correlations, cross-loading onto the wrong factor, confusing wording flagged by respondents, should be revised or removed before your final version goes out.

Step 7: Prepare Your Documentation for Defense

By the time you reach your proposal or final defense, you should be able to present, clearly and without hesitation:

• Your construct definitions and how each maps to specific items
• Whether you adopted, adapted, or developed the instrument, and your rationale
• Your content validation process, including expert credentials and CVI results
• Your translation and back-translation process, if applicable
• Your pilot test results, including reliability coefficients per subscale
• Any items removed or revised, and why

Panels respond well to students who can narrate this process as a coherent story rather than presenting a finished questionnaire with no visible trail of how it got there. The instrument isn’t just a data collection tool; it’s evidence of your methodological rigor, and presenting the process behind it is often what separates a smooth defense from a defensive one.

Common Mistakes That Trigger Panel Pushback

Skipping expert validation entirely. This is the single most common gap we see, and it’s also the easiest to fix before your proposal defense, not after.

Using an instrument from a foreign study without any adaptation or re-validation. Cultural and linguistic context matters more than students often expect, particularly for constructs tied to workplace norms, social relationships, or values.

Treating pilot testing as optional or skipping straight to full data collection. If your reliability results come back poor after your main data collection is already done, you have far fewer options for fixing the problem than if you’d caught it during piloting.

Vague or circular construct definitions. If your definition of the construct is essentially a restatement of the construct’s name, your panel will ask you to go deeper, and it’s better to have that depth ready before the question comes.

No clear connection between items and dimensions. Panels often ask which items measure which dimension of your construct. If you can’t answer quickly, it suggests the instrument wasn’t built from a clear conceptual framework in the first place.

The Bottom Line

A survey instrument that survives panel scrutiny isn’t the result of clever wording. It’s the result of a documented process: clear construct definitions, a deliberate choice between adopting, adapting, or developing, expert content validation, careful translation where needed, and a real pilot test before you commit to full data collection. Students who can walk their panel through that process, step by step, tend to have a noticeably smoother defense than those who show up with only a finished questionnaire and no visible trail behind it.

If you’re building or revising a survey instrument and want a second opinion before you finalize it, or help running your pilot reliability and validity analysis, that’s exactly the kind of question StatAce exists to help with.

---

Building a survey instrument for your thesis and not sure it will hold up under questioning? Reach out to StatAce; we help Philippine graduate students and researchers design and validate instruments before data collection begins.