Skip to content

FAQ: UT Core Framework Overview

Jump to bottom
Ulrond edited this page Feb 3, 2025 · 4 revisions

Excerpt from 3.2. Standards: Requirements for Building Testing Suites

  • L1/L2 testing suites will use the ut-core testing framework.
  • L3/L4 testing suites will incorporate the python-raft infrastructure and the ut-raft framework while continuing to utilize ut-core as required.
  • The testing system must adhere to the guidelines outlined in the Setup/Layout Guide.
  • Developers are expected to familiarize themselves with the ut-raft classes, documented in the ut-raft wiki.
  • Examples of python-raft configuration and setup can be found in the aforementioned resources.

Levels of Test

The levels of testing to be implemented are described in detail here:
Standards: Levels of Test for Vendor Layer

Testing Frameworks

Actors Involved

  • Automation
  • QA Team
  • Engineering Team
  • DUT (Device Under Test)

Engineering Testing

  • Engineers and third-party developers will use Levels 1 to 3 to validate and test changes to individual components or groups of components.
  • Level 4 testing expands to multi-component testing, ensuring end-to-end validation.
  • Pre-commit testing is covered under Levels 1 - 3, requiring manual review of results but not necessarily automation.
Test Suite Actors
Level 1 - 3 Component Tests Engineers
L4 System Performance Engineers
L4 Ad-hoc Analysis Engineers
L4 Smoke Testing Layer Release Engineers, QA Team
L4 System Interface Testing Engineers
L4 Deep Dive Testing Engineers

QA Testing

  • QA primarily focuses on L4 Smoke Testing to ensure compatibility throughout the layer release cycle.
  • The full stack image includes Vendor, MW, and Application Layers, enabling test binaries to be run using previous versions to identify regressions.

System Architecture Overview

Testing System Flow

graph LR
    subgraph Actors
        subgraph "Python Raft / UT-Raft"
            Python_Raft --> device(DUT)
        end
        subgraph "XTS Tools"
            xTS --> Python_Raft
            xTS <--> allocator[xTS Allocator]
            allocator --> rackConfig
            rackConfig --> Python_Raft
        end
        subgraph "Automation"
            auto1[Github Actions / Review Approval] --> xTS
        end
        subgraph "QA Team"
            QASmoke[L4 Smoke Testing] --> xTS
        end
        subgraph "Engineer - Component / Layer / Vendor"
            L4Smoke[L4 Testing] --> Python_Raft
            EngineerComponentTesting[L3 Component Testing] --> Python_Raft
            EngineerComponentTesting --> device
            L4Smoke --> localRackConfig
            EngineerComponentTesting --> localRackConfig
            localRackConfig --> Python_Raft
        end
    end
Loading

Host Machine (High-Level Tests)

L3 Component Stimulus Testing and L4 System Testing are executed from the Host Machine, utilizing the ut-raft framework to extend python-raft capabilities.

graph LR
    subgraph "Host Machine (High Level Tests)"
        deviceConfig --> Python_Test
        rackConfig --> Python_Test
        Python_Test[Python Test] --> ut_raft
        ut_raft --> python_raft
        python_raft --> console
        console --> DUT
        python_raft --> webPageControl --> DUT
        python_raft --> OutboundClient --> DUT
    end
    subgraph "Rack Slot"
        python_raft --> PowerSwitch --> DUT
        python_raft --> IR --> DUT
        python_raft --> CECAdaptor --> DUT
        python_raft --> VideoCapture --> DUT
        VideoCapture --> HostHDD
    end
Loading

DUT Target Architecture

Python Raft connects to the DUT via a console session, ensuring platform-independent testing with automated configurations.

graph LR
    subgraph "DUT (Target)"
        console <--> test_binary(Test Code)
        platform_profile --> test_binary
		test_binary --> state_machine
		state_machine --> code(Code Under Test)
        test_binary --> ut_core
        ut_core --> ut_control
        webSocket --> ut_control
		ut_control --> state_machine
    end
Loading

Summary

  • The testing system is divided into multiple levels, utilizing ut-core, python-raft, and ut-raft for different types of testing.
  • Engineering Teams perform pre-commit testing at Levels 1 - 3, while QA Teams focus on L4 Smoke Testing.
  • Automation integrates with testing through GitHub Actions and other tools.
  • DUT Target Testing ensures platform-independent configuration and verification of vendor-layer compatibility.

This document provides a graphical overview of the testing system, ensuring that all stakeholders have a clear understanding of the testing flow and responsibilities.

Clone this wiki locally