Dummy data files with date columns

[iaindillingham]

It's now possible to pass a dummy data file when generating a cohort, avoiding some clunkiness associated with the expectations framework.

opensafely/documentation#499 describes how a researcher passed a dummy data file with a date column. However, the corresponding column in the study definition was the result of calling patients.categorised_as, meaning that if the dummy data file hadn't been passed, then the date column would have been a string column.

When verifying dummy data files, cohort-extractor assumes that a date column (query_args["column_type"] == "date") will be associated with a date_format argument. This assumption doesn't hold, here, so cohort-extractor raised the following error:

Traceback (most recent call last):
  File "/usr/local/bin/cohortextractor", line 33, in <module>
    sys.exit(load_entry_point('opensafely-cohort-extractor', 'console_scripts', 'cohortextractor')())
  File "/app/cohortextractor/cohortextractor.py", line 668, in main
    generate_cohort(
  File "/app/cohortextractor/cohortextractor.py", line 151, in generate_cohort
    _generate_cohort(
  File "/app/cohortextractor/cohortextractor.py", line 202, in _generate_cohort
    study.to_file(
  File "/app/cohortextractor/study_definition.py", line 92, in to_file
    validate_dummy_data(self.covariate_definitions, dummy_data_file)
  File "/app/cohortextractor/validate_dummy_data.py", line 31, in validate_dummy_data
    validator = get_binary_validator(col_name, query_args)
  File "/app/cohortextractor/validate_dummy_data.py", line 174, in get_binary_validator
    if query_args["date_format"] == "YYYY":
KeyError: 'date_format'

Here's the line, with some context:

cohort-extractor/cohortextractor/validate_dummy_data.py

Lines 172 to 175 in 1514a7a

	column_type = query_args["column_type"]
	if column_type == "date":
	if query_args["date_format"] == "YYYY":
	return date_validator_year

The line and the KeyError aren't especially informative. Instead, we could...

• return date_validator_day (we might need to replace the call to isinstance with a call to is_datetime64_any_dtype)
• raise a more informative error
• something else 🤔

[evansd]

Ah, I suspect this is related to the fact that categorised_as() expressions don't necessarily return strings. They can return any type (including dates) depending on the type of the supplied categories. See:

cohort-extractor/cohortextractor/process_covariate_definitions.py

Lines 432 to 462 in 1514a7a

	def type_of_categorised_as(self, category_definitions, **kwargs):
	return self._infer_type_from_categories(category_definitions)
	def _infer_type_from_categories(self, category_definitions):
	# Convert date-like strings to dates
	categories = [
	datetime.date.fromisoformat(v)
	if isinstance(v, str) and re.match(r"\d\d\d\d-\d\d-\d\d", v)
	else v
	for v in category_definitions.keys()
	]
	first_type = type(categories[0])
	for other in categories[1:]:
	if type(other) != first_type:
	raise ValueError(
	f"Categories must all be the same type, found {first_type} "
	f"and {type(other)}"
	)
	if first_type is int:
	if set(categories) == {0, 1}:
	return "bool"
	else:
	return "int"
	elif first_type is float:
	return "float"
	elif first_type is str:
	return "str"
	elif first_type is datetime.date:
	return "date"
	else:
	raise ValueError(f"Unhandled category type: {first_type}")

So I wonder if the simple solution here is for categorised_as expressions of type date to automatically be assigned a date_format="YYYY-MM-DD" argument so they function as well-behaved dates elsewhere in the system.

I guess this would be done in an extra function call after add_column_types():

cohort-extractor/cohortextractor/process_covariate_definitions.py

Line 22 in 1514a7a

covariate_definitions = add_column_types(covariate_definitions)