Metrology Terminology: Accuracy, Precision, Resolution

Accuracy vs Precision vs Resolution

In metrology, using the correct terminology can be critical in getting the measurement information required. Three terms that are often incorrectly used interchangeably are accuracy, precision, and resolution.

When speaking about the accuracy of a measurement, you are referring to the data’s correctness. This means that an accurate instrument would provide measurements closest to the actual value or standard. Precision, on the other hand, is the ability of a device to provide you with consecutive measurements that are close to one another. An instrument can give extremely precise measurements that are not at all correct. Precision is directly related to the repeatability and reproducibility of your measurements.

 

AccuracyAccuracy

PrecisionPrecision

Accuracy & PrecisionAccuracy/Precision

 

Accurate But Not Precise?

If an instrument were accurate but not precise, data collected using the device would hover close to the true value, but there will be disagreement between the individual readings. Conversely, a device that is precise but not accurate will give you very repeatable measurements that are not necessarily correct. Quality measurement equipment should give you readings that are both accurate and precise, meaning the measurement data is close to the true value and is highly repeatable and reproducible.

Another term often confused with accuracy and precision is resolution, which is the smallest change that can be measured with the instrument. Also referred to the least significant digit or LSD. In scales and balances this would be referred to as divisions or counts. For example, the smallest unit change you can see in a floor scale might be a pound, while a balance could measure weight in milligrams. The floor scale has comparably low resolution to the balance, but for the application it is not necessary to know the weight of a pallet down to the milligram.

 

Resolution Misconceptions

A common misconception is that a device with high resolution is very precise and accurate. This isn’t necessarily true. Though resolution does directly relate to precision, it does not guarantee accuracy. A balance could have very high resolution, while giving inaccurate or unreliable measurements. Attempting to alter the settings of a scale to make it have a higher resolution than for which it has been NTEP approved can have a serious impact on its performance. Referred to as stretched resolution, these scales are set up in such a way that the parameters are beyond those stated on their Certificate of Conformance (COC). Because the scale is not intended to be used in this way, you cannot ensure that it will stay within calibration for any length of time, and users should know that altering scale in this way does not improve accuracy.

It’s critical when installing and calibrating scales, that each component of the weighing system is evaluated to make sure it will meet your accuracy, precision and resolution requirements. This is also generally true for any piece measurement equipment; ensuring that it meets the needs of your specific application is essential. If you are unsure about what level of accuracy, precision, or resolution that you need, our team of measurement experts are here to help. Contact us today to find the right equipment for your application.

 

How Can Cross Help?

In the field of metrology, there are many standards to follow to ensure reliable calibration or inspection. At Cross Precision Measurement, we are an ISO 17025 accredited calibration laboratory that adheres to standards set out by NIST (HB44), the National Conference on Weights and Measures (NCWM), the National Type Evaluation Program (NTEP), A2LA, and other relevant standards. With our experience and expertise, we can ensure you’re getting reliable, repeatable metrology results.

Since 1939, our Precision Measurement group has provided industry-leading technical expertise coupled with world-class customer service to manufacturing and other industrial operations in need of onsite calibration, product inspection, precision measurement equipment, and more. Our precision measurement and inspection teams have the knowledge and resources to help you reduce risk, improve quality, and increase efficiency throughout your entire process. With our extensive network of calibration labs across the Southeast and Midwest, we can ensure you’re getting the metrology services you need. Contact Cross Precision Measurement today

Latest Cross Company Resources

This automated palletizing cell improves stack consistency, reduces labor, and lowers the risk of injury. Watch the video to see how! CrossCo Automation.
October 2, 2025
Cross Logo
Dr. Daniel Moore is an Assistant Professor in the Ophthalmology and Visual Sciences Department at a major research university. He specializes in glaucoma, an eye disease which is due to several different factors.
September 30, 2025
Cross Logo
Historically, Lindy’s relied on small thermometers mounted on support columns to keep track of the temperature in their cold storage. Several times a day, someone had to walk around the refrigerated warehouse, collecting data by hand and then manually entering the information into the company’s quality system. Not only was
September 30, 2025
Cross Logo
KapStone Paper and Packaging makes unbleached kraft paper and high performance lightweight liner board. Their Roanoke Rapids, North Carolina plant, set on the banks of the Roanoke River, runs 24/7 with approximately 430 employees producing over 470,000 tons of paper per year. The Fiber Supply department works with local vendors
September 30, 2025
Cross Logo
Sutherland Products is a boutique manufacturer of commercial and household cleaning products located in Mayodan, North Carolina. The company is best known for its Charlie’s Soap brand of laundry detergents and cleaning products, which combine effective cleaning and low environmental impact. From its origins in the Piedmont Triad area, the
September 30, 2025
Cross Logo
Commonwealth recently launched their Organisational Excellence Programme, or OEP, a continuous improvement initiative based on LEAN principles. Said John McPherson, Commonwealth’s Secondary Production Manager, “OEP is all about driving performance through increasing Overall Equipment Effectiveness (OEE) and minimizing waste.”
September 30, 2025
Cross Logo

See how our team can help improve efficiency, reduce risk, and raise quality in your process.

Contact our Team

Hang Tight! We're Searching... Searching... Searching...

We’re looking through thousands of pages to find the most relevant information.

In the meantime, enjoy these fun facts…

Did you know… Cross Company is an ESOP (Employee Stock Ownership Plan). Our ESOP started in 1979 and as of 2006, we are 100% employee-owned! Learn more about our ESOP and how that benefits both team members and our customers.
Did you know... the precision measurement group at Cross was founded in 1939 by our current CEO's grandfather, Jim King. That's a whole lot of calibration!
Did you know... A fingerprint weighs about 50 micrograms. We know, we weighed it! The residue left from a finger can actually make a difference in weight results which is why we wear gloves when we calibrate weights. For reference, a sheet of paper is about 4.5 grams, that’s 4.5 million micrograms.
Did you know… Cross Company has grown significantly since our start in 1954. Over the years we've acquired 26 companies! Today, our five groups have expertise in everything from industrial automation to precision measurement, and industry knowledge going all the way back to 1939.