System Logs Dataset Specification
Audit Log
The "Audit Log" dataset serves as a detailed record of actions and changes made to entities within the system. It tracks various operations such as creation, modification, deletion, and viewing of records. Each entry in the log includes the entity’s unique identifier, the type of action taken, the user responsible for the action, and a timestamp of when the action occurred. The dataset provides transparency and traceability for all transactions, making it an essential component for ensuring data integrity, security, and compliance within the system.
Table Structure: Audit Log
Table Name | Column Name | Is Nullable | Data Type | Description |
Audit_log | Entity_ID | YES | STRING | Unique identifier for the entity or record being audited. |
Audit_log | Entity_Name | YES | STRING | Name of the entity or record type that is being logged in the audit. |
Audit_log | Action | YES | STRING | The specific action taken on the entity, such as create, update, delete, or view. |
Audit_log | Record_Last_Modified_Timestamp | YES | TIMESTAMP | Timestamp indicating the last time the record was modified or updated. |
Audit_log | Transaction_User_ID | YES | STRING | The ID of the user who performed the action or transaction being logged. |
Audit_log | Payload | YES | STRING | The detailed data or content related to the action performed, often containing before-and-after states or changes made. |
Extract Runs Log
The "Extract Runs Log" dataset provides detailed information about the execution of data extraction processes within the system. It records metadata about each extract run, including the table being processed, the time intervals for extraction, and the status of the run. This dataset is critical for monitoring and troubleshooting data pipeline operations.
Table Structure: Extract Runs Log
Table Name | Column Name | Is Nullable | Data Type | Description |
extracts_run_logs | Dag_Run_Id | NO | STRING(100) | The unique identifier for the Directed Acyclic Graph (DAG) run in the data pipeline. |
extracts_run_logs | Timestamp_Id | NO | TIMESTAMP | A timestamp identifier associated with this extract run. |
extracts_run_logs | Table_Name | NO | STRING(100) | The name of the table being extracted or processed. |
extracts_run_logs | Start_Interval | YES | TIMESTAMP | The start time of the interval for which data is being extracted. |
extracts_run_logs | End_Interval | YES | TIMESTAMP | The end time of the interval for which data is being extracted. |
extracts_run_logs | Grace_Period_Min | YES | INT64 | The grace period in minutes allowed for late-arriving data. |
extracts_run_logs | Status | NO | STRING(50) | The current status of the extract run (e.g., success, failed, in progress). |
extracts_run_logs | Number_Of_Files | YES | INT64 | The number of files processed during this extract run. |
extracts_run_logs | Number_Of_Entries | YES | INT64 | The number of data entries or rows processed during this extract run. |
extracts_run_logs | Last_Step_On_Failed | YES | STRING(100) | The last step that was executed before a failure occurred, if applicable. |
extracts_run_logs | Error_Message | YES | STRING(250) | Any error message associated with a failed extract run. |
extracts_run_logs | Is_Shadow | YES | BOOL | Indicates whether this is a shadow run (test run) or a production run. |
extracts_run_logs | Create_Timestamp | NO | TIMESTAMP | The timestamp when this extract run log entry was created. |