Techno Blender
Digitally Yours.

One Simple Methodology to Turn a Subjective Concept into an Objective Metric | by Jordan G. | Sep, 2022

0 206


Tap into the experts in your organization to create a source of truth and train a model against it

Working in a business team, I am often asked to quantify a subjective concept for which there is no broadly accepted definition. A typical question looks like: “how many of our stores are successful?” or “how many loyal customers do we have?”

Photo by Greyson Joralemon on Unsplash

The crux of the issue is that everyone has their own ‘theoretical’ definition for these subjective words (and believe their understanding is shared across the company). So when asked such a question, the main challenge is to actually align all the different point of views to actually come up with an objective definition.

To give a concrete example — let’s say that you work for an important supermarket chain, and have been asked “how many stores are successful?”

You could look into ‘profit’ and start thinking about ranking the different stores based on profitability, but then:

  • Assuming that in your industry profit is a function of the size — would a small store with not a lot of profit be less successful?
  • What about a store that just launched and is showing tremendous growth but won’t be profitable until next year? Or on the contrary, what about a store with large profits but a negative growth? Which one would you flag as successful?
  • What about a very profitable store that has been gathering an increasing number of negative customer reviews? Or that has a very high employee turnover?

As we see above, subjective concepts like “successful” or “loyal” can have different definitions across your organization. It is important and can be quite hard to align these definitions into one solid metric.

There are different ways to go about this problem, but one that I find particularly elegant is the following: Build a source of truth in collaboration with a few stakeholders.

Basically — ask several internal experts to review and classify a representative sample of your population, and from there fit a model against the manual classifications. This will allow you to understand what is the ‘data’ definition that best matches the subjective concept you are trying to define.

  • This approach enables alignment inside your organization as you bring in stakeholders from different teams with different point of views.
  • It can also create meaningful discussions and unlock new learnings, especially when two different experts have different classifications of the same record — this will help you and them understand each other’s rationale.
  • Pull a list of records that are representative of your whole population
An example of a sample (by author)
  • Ask a few experts internally to help manually classify the different records and review the classifications with them to come up with a final rating
The reviewed sample (by author)
  • Congratulations! You now have a source of truth for your new metric, and this is now a much more simple classification problem. You can start fitting a model against the final ratings, and adjust it based on your needs (i.e. how you balance explainability vs complexity or accuracy vs precision, or else).
  • Once you have a 1st definition, present it to your audience, and show them what that would mean to the business (with some concrete examples). Iterate as needed.

The methodology presented above is one of many methodologies that you can use when presented with these kind of questions. It is not a panacea, and it does require you to find some experts willing to spend time transferring the knowledge they have been internalizing for years, but it can work pretty well when you are working on projects requiring a lot of alignment and/or on projects where you are stuck and you don’t even know where to begin.

If you had fun see my other articles:


Tap into the experts in your organization to create a source of truth and train a model against it

Working in a business team, I am often asked to quantify a subjective concept for which there is no broadly accepted definition. A typical question looks like: “how many of our stores are successful?” or “how many loyal customers do we have?”

a ball pit of differently colored balls
Photo by Greyson Joralemon on Unsplash

The crux of the issue is that everyone has their own ‘theoretical’ definition for these subjective words (and believe their understanding is shared across the company). So when asked such a question, the main challenge is to actually align all the different point of views to actually come up with an objective definition.

To give a concrete example — let’s say that you work for an important supermarket chain, and have been asked “how many stores are successful?”

You could look into ‘profit’ and start thinking about ranking the different stores based on profitability, but then:

  • Assuming that in your industry profit is a function of the size — would a small store with not a lot of profit be less successful?
  • What about a store that just launched and is showing tremendous growth but won’t be profitable until next year? Or on the contrary, what about a store with large profits but a negative growth? Which one would you flag as successful?
  • What about a very profitable store that has been gathering an increasing number of negative customer reviews? Or that has a very high employee turnover?

As we see above, subjective concepts like “successful” or “loyal” can have different definitions across your organization. It is important and can be quite hard to align these definitions into one solid metric.

There are different ways to go about this problem, but one that I find particularly elegant is the following: Build a source of truth in collaboration with a few stakeholders.

Basically — ask several internal experts to review and classify a representative sample of your population, and from there fit a model against the manual classifications. This will allow you to understand what is the ‘data’ definition that best matches the subjective concept you are trying to define.

  • This approach enables alignment inside your organization as you bring in stakeholders from different teams with different point of views.
  • It can also create meaningful discussions and unlock new learnings, especially when two different experts have different classifications of the same record — this will help you and them understand each other’s rationale.
  • Pull a list of records that are representative of your whole population
An example of a sample (by author)
  • Ask a few experts internally to help manually classify the different records and review the classifications with them to come up with a final rating
The reviewed sample (by author)
  • Congratulations! You now have a source of truth for your new metric, and this is now a much more simple classification problem. You can start fitting a model against the final ratings, and adjust it based on your needs (i.e. how you balance explainability vs complexity or accuracy vs precision, or else).
  • Once you have a 1st definition, present it to your audience, and show them what that would mean to the business (with some concrete examples). Iterate as needed.

The methodology presented above is one of many methodologies that you can use when presented with these kind of questions. It is not a panacea, and it does require you to find some experts willing to spend time transferring the knowledge they have been internalizing for years, but it can work pretty well when you are working on projects requiring a lot of alignment and/or on projects where you are stuck and you don’t even know where to begin.

If you had fun see my other articles:

FOLLOW US ON GOOGLE NEWS

Read original article here

Denial of responsibility! Techno Blender is an automatic aggregator of the all world’s media. In each content, the hyperlink to the primary source is specified. All trademarks belong to their rightful owners, all materials to their authors. If you are the owner of the content and do not want us to publish your materials, please contact us by email – [email protected]. The content will be deleted within 24 hours.

Leave a comment