Skip to content

Files

Latest commit

547f6e1 · Apr 17, 2021

History

History
This branch is 423 commits behind ARM-software/EndpointAI:master.

Arm-2D

README

NOTE:

  1. This library is a research project used to explore and demonstrate the possibilities of delivering smart-phone modern graphic user interface using low-cost and resource constraint micro-controllers. It is not a committed product from Arm, and the quality of the service is not validated with sophisticated tests but some functional tests.
  2. The library name, i.e. Arm-2D, is a temporary term and might be subject to change in the future. The term "the Library" used in this documents refers to the Arm-2D library unless state otherwise.

The Purpose of Branches In This Repository

Branch Name Description Note
master/main The main branch of the repository. It contains the Arm-2D library and simple examples.
main-arm-2d-developing The development branch for the main branch. It will be merged into main branch.
main-arm-2d-more-examples It has the same content as main branch but with additional examples.
The purpose of this branch is adding more examples for people who want to try while still keeping the main branch simple and small.
NOTE: This branch will not modify anything in the Arm-2D library but only focuse on examples.
main-arm-2d-more-example-developing It is the development branch for the main-arm-2d-more-examples-developing branch. The main branch will be merged into this one which will be merged into the main-arm-2d-more-examples branch later.

Features of the Arm-2D Library

In this release ( ver0.9.0 )

The Arm-2D library provides Low-Level 2D Image Processing Services that are mainly used in Display system. Those servers include but not limited to:

  • Alpha-Blending
    • With or without Colour-Masking
  • Image Copy / Texture Paving
    • With or without Colour-Masking
    • Four mirroring mode: None, X-mirroring, Y-mirroring and XY-mirroring
  • Colour format conversions
    • RGB565 and RGB888
    • Generic RGB16 and RGB32
  • Region/Window Clipping
  • Generic Partial Frame-buffer (PFB) Support
    • Transparent for upper layer software/GUI services
    • Easy to implement
    • No limitation on target screen resolution
    • No limitation on PFB size and shape (it could be line or cube with any size)
  • Unified and User Friendly Programmers' Mode
    • APIs could be used in Synchronous manner ( Classic Blocking code ) and/or Asynchronous manner ( Event-Driven )
    • Support both bare-metal and RTOS
    • Ultra small memory footprint

Planned in the Future

Following features are planned and to be introduced in the near future:

  • Alpha (bitmap) Masking schemes
    • New APIs will be added for Copy, Paving and Alpha-blending
  • Colours that contain Alpha-channel, i.e. RGBA8888, RGBA5651
  • Rotations Algorithms
  • Image Filters, e.g. Anti-aliasing algorithms
  • Zooming/Stretching Algorithms

1 Introduction

1.1 The Background

With more and more smart IoT edge devices introduced to our daily lives, people who are used to the smart-phone like a graphic user interface (GUI) want to have the same user experience when using those micro-controller-based products. This trend has been long observed and understood by Arm's leading eco-partners.

As a result, many silicon vendors introduce dedicated hardware accelerators into their microcontroller products to help 2D image processing. Meanwhile, GUI service providers also update their product lines to target micro-controller based graphic user interface applications. Many open-source embedded GUI stacks are hot on GitHub, e.g. LVGL.

In fact, using GUI with microcontrollers isn't new at all. Still, until the recent rise of IoT and AI, people have found that the simple and monotonous graphical interface in the past is really unbearable. The pursuit of user experience similar to that of a smartphone has become a basic requirement for products.

On the other hand, seemingly complicated graphical interfaces often only require simple texture paving. Even the so-called transparency effects are not unaffordable for microcontrollers that often run under tens of MHz or even hundreds of MHz.

Technologies used for 2D graphics have been matured as early as the era of 8-bit Gaming Console. As 8-bit 6502 can achieve fancy graphics effects, why can't the most advanced Cortex-M processor of the day?

Figure 1-1 2D Technologies used in Super-Mario-Brothers on NES

image-20210318235155494

1.2 The Problems in current solutions

As shown in Figure 1-2, the Linux system has a complete ecosystem from GPU drivers to GUI software services. Compared with it, the resource-constrained embedded system is obviously missing a lot in the ecosystem. To make up for this missing link, the concept of Arm-2D was introduced.

Figure 1-2 Ecosystem Comparison between Rich Embedded and Constraint Embedded System in GUI

image-20210318235815106

When we look at the conventional GUI graphics architecture in today's embedded ecosystem (as shown in Figure 1-3 ), we will clearly observe 4 layers: the application and design layer, the GUI software service layer, the rendering layer and hardware driver layer.

Arm-2D focuses on accelerating the low-level 2D image processing, and will not touch any part of the GUI software service, so the library will not compete with the GUI service providers in the Arm ecosystem. In fact, because Arm has proposed a unified set of low-level acceleration APIs, a full ecological level of cooperation can be quickly established between chip manufacturers that provide hardware accelerators and software providers that provide GUI services. Everyone can concentrate on their own works: For example, chip manufacturers can ensure that they receive a wide range of software support by adding drivers for their dedicated 2D accelerators following the Arm-2D standard, and GUI providers only need to build their GUI stack upon Arm-2D APIs; hence a wide range of device support is ensured.

Figure 1-3 The Hierarchy of a Typical Embedded GUI System.

1.3 The Platform

The library is targeting ALL Cortex-M processors with/without various hardware 2D image accelerators:

  • Armv6-M processors: Cortex-M0/M0+/M1
  • Armv7-M processors: Cortex-M3/M4/M7
  • Armv8-M processors: Cortex-M23/M33/M35P
  • Armv8.1-M processors: Cortex-M55

The library is designed with ACI (Arm Custom Instructions) in mind. Specific accelerations using user-defined instructions could be easily integrated into the library without modifying upper-layer software.

The library is designed with various 2D image accelerator in mind. The Support for those accelerators could be easily added into the library without modifying upper-layer software.

The library is designed with a deep constraint environment in mind. For Cortex-M processors with 4K~32K RAM that traditionally cannot afford a full-frame-buffer, it introduces a feature called Generic Partial Frame-buffer which enables those existing MCUs to exchange time for space and have a fancy graphical user interface whilst still have a decent frame-rate.

1.4 Dependency

  • The library depends on CMSIS 5.4.0 and above.
  • The library is developed with the C11 standard and depends on some widely adopted GCC extensions.
    • See section 3.2 for details.
  • The library should be compiled with Arm Compiler 5, Arm Compiler 6, GCC, LLVM and IAR
    • See section 5 for details.

1.5 Examples and Benchmark

1.5.1 Example Summary

Example Description Folder Note
Alpha-Blending It is an ALL-IN-ONE example that demonstrates almost all the features provided by the library. examples/alpha_blending Used as benchmark.
Partial-Frame-buffer It delivers the same visual effects as Alpha-blending example but using Partial-Frame-buffer. It can be used as a template or reference code for programmers who want to implement a graphical user interface on an MCU with small RAM. In this example, 16*16 FPB (512Bytes) is used, and the total system RAM usage is less than 2.5KByte (including stack, heap and FPB). examples/partial_frame_buffer.

1.5.2 Benchmark

Since there is no public benchmark available for micro-controllers, we decide to overcome this problem with the following methods and considerations:

  • Choose the widely used algorithms in embedded GUI as the body of the benchmark
    • Alpha-blending
    • Image Copy
    • Texture Paving
  • Simulate a typical application scenario with sufficient complexity
    • Background with Texture paving (switching different mirroring modes every 4 second)
    • Foreground picture
    • Two constructed layers for alpha-blending and texture paving
    • Moving icons
    • Spinning busy wheel
  • Choose a typical low-cost LCD resolution 320*240 in RGB565
  • Let those layers float with different angles and speed to cover a sufficient number of conditions.
  • Record the cycle count used for blending one frame and run 1000 iterations (frames).

Figure 1-4 A snapshot of alpha-blending demos running on MPS3 platform

Alpha-blending

  • Use the average cycle count in 1000 iterations as benchmark score.

    • Based on that, for typical embedded application requirement, we derive a more meaningful metrics called the Minimal Frequency Required for 30 FPS (MHz) as shown in Figure 1-5.

Figure 1-5 Performance Comparison among some Cortex-M processors

image-20210318225839820

2 Folder Hierarchy

Folder and File Type Description
Library Folder This folder contains the source files and header files of the library.
Documents Folder This folder contains all the documents.
Examples Folder This folder contains all the example projects.
README .md The README.md you are currently reading.
how_to_deploy_the_arm_2d_library .md A step by step guidance helping you to deploy the Arm-2D library to your projects.
LICENSE The Apache 2.0 License

3 Tips For Exploring the Library

3.1 "I am a library user, I only care about how to use the library"

  • For library users, ALL useful information, i.e. type definitions, macros, prototypes of functions etc., are stored in header files which have NO double under-scope as their prefixes. We call those header files the PUBLIC HEADER FILES.
  • Please ONLY use APIs, macros and types that are defined in the public header files.

Figure 3-1 Private and Public Files

image-20210317181453270

  • Any symbol, e.g. file name, function name, macro name, type name etc., having a double under-scope as the prefix is considered as PRIVATE to the library. You should save your time from touching them.

  • The library is designed with the philosophy that Users are free to use anything in public header files and should not touch anything marked implicitly or explicitly as private.

  • Despite which processor you use, during the compilation, all C source files are safe to be added to the compilation (and we highly recommend you to do this for simplicity reason). For example, when you use Cortex-M4, which doesn't support Helium extension (introduced by Armv8.1-M architecture and first implemented by the Cortex-M55 processor), it is OK to include "arm_2d_helium.c" in the compilation process, as the C source files are constructed with environment detection pre-processing mechanisms.

  • In your application, only including "arm_2d.h" is sufficient to get all the services and APIs ready for you.

  • Make sure that the library is initialised by calling arm_2d_init() before using any of the services.

    NOTE:

    1. arm_2d_init() is a function-like macro that initialises different parts of the library depending on the feature configuration macros.
    2. Feature configuration macros are checked by "arm_2d_feature.h". For the current stage of the library, please DO NOT override those feature configuration macros.
#undef __ARM_2D_HAS_HELIUM__
#if defined (ARM_MATH_HELIUM) || defined(ARM_MATH_MVEF) || defined(ARM_MATH_MVEI)
#   define __ARM_2D_HAS_HELIUM__                        1
#endif

#ifndef __ARM_2D_HAS_CDE__
#   define __ARM_2D_HAS_CDE__                           0
#endif

#ifndef __ARM_2D_HAS_HW_ACC__
#   define __ARM_2D_HAS_HW_ACC__                        0
#endif
#if defined(__ARM_2D_HAS_HW_ACC__) && __ARM_2D_HAS_HW_ACC__
#   if defined(__ARM_2D_HAS_ASYNC__) && !__ARM_2D_HAS_ASYNC__
#       warning As long as __ARM_2D_HAS_HW_ACC__ is set to 1,\
 __ARM_2D_HAS_ASYNC__ is forced to 1. Since you set __ARM_2D_HAS_ASYNC__ to\
 0, please remove your macro definition for __ARM_2D_HAS_ASYNC__ to avoid this\
 warning.
#   endif
#   undef __ARM_2D_HAS_ASYNC__
#   define __ARM_2D_HAS_ASYNC__                         1
#endif

#ifndef __ARM_2D_HAS_ASYNC__
#   define __ARM_2D_HAS_ASYNC__                         1
#endif
#if defined(__ARM_2D_HAS_ASYNC__) &&  __ARM_2D_HAS_ASYNC__
#   if  !defined(__ARM_2D_DEFAULT_SUB_TASK_POOL_SIZE) ||                        \
        __ARM_2D_DEFAULT_SUB_TASK_POOL_SIZE < 4
#       define __ARM_2D_DEFAULT_SUB_TASK_POOL_SIZE      4
#   endif
#endif

/*============================ MACROFIED FUNCTIONS ===========================*/

#if defined(__ARM_2D_HAS_CDE__) && !__ARM_2D_HAS_CDE__
#   define __arm_2d_cde_init()
#endif

#if defined(__ARM_2D_HAS_HELIUM__) && !__ARM_2D_HAS_HELIUM__
#   define __arm_2d_helium_init()
#endif

#if defined(__ARM_2D_HAS_HW_ACC__) && !__ARM_2D_HAS_HW_ACC__
#   define __arm_2d_acc_init()
#endif

#if defined(__ARM_2D_HAS_ASYNC__) && !__ARM_2D_HAS_ASYNC__
#   define __arm_2d_async_init()
#endif


#undef arm_2d_init
#define arm_2d_init()                                                           \
        do {                                                                    \
            __arm_2d_init();                                                    \
            __arm_2d_async_init();                                              \
            __arm_2d_helium_init();                                             \
            __arm_2d_cde_init();                                                \
            __arm_2d_acc_init();                                                \
        } while(0)

3.2 "I am interested in the implementation"

  • We apologise that at the current stage (it's the early stage, as you can see), there is no sufficient guidance or documents about:
    • How the library is implemented
    • How to contribute
    • How to add new features
    • What's the design principles behind the code
    • What's the structure of the design in details
  • Some design considerations:
    • The library is supposed to support Arm Compiler 5/6, GCC, LLVM and IAR. So far, ONLY Arm Compiler 6 is validated.
    • The library is supposed to support ALL Cortex-M processors. There should be no problem for working with existing Cortex-M processors, i.e. Cortex-M0/M0+/M1/M3/M4/M7/M23/M33/M35P/M55. If you find any problems, please feel free to raise an issue.
    • The library is designed with some OOPC (Object-Oriented Programming with ANSI-C) methodologies. And the bottom line is that any methods and tricks adopted in this library should come with no or very little cost.
  • This library is compliant with C11 standard and uses some widely accepted GCC extensions:
  • Some of the definitions are written with the support of the Microsoft Extensions in mind ( -fms-extensions ), but the library never depends on it. This means that if programmers enable the support of the Microsoft Extensions in their project, they can benefit from it.
  • This library follows "Using Extensions to replace Modifications" principle
    • Keywords __WEAK and __OVERRIDE_WEAK are introduced for default functions and extensions; it is similar to the concept of "virtual functions" and "override functions" in C#.
      • arm_2d_async.c is used to override some infrastructure functions in arm_2d.c to support asynchronous mode in the programmers' mode.
      • arm_2d_helium.c is used to override some default software algorithm implementations across the library.
    • Supports for hardware accelerators (both from Arm and 3rd-parties) should be added in the same manner in the future.
      • Override a table called __ARM_2D_IO_TABLE[] originally defined in arm_2d_op_table.c to add your own version of algorithms and hardware accelerations.
//! \name Operation Low Level IO Index
//! @{
enum {
    /*------------ cmsisi-2d operation idx begin ------------*/
    __ARM_2D_IO_NONE = -1,
    
    __ARM_2D_IO_COPY,
    __ARM_2D_IO_FILL,
    ...
    __ARM_2D_IO_ALPHA_BLENDING,
    ...
    __ARM_2D_IO_COLOUR_CONVERT_TO_RGB565,
    ...
    
    /*------------ cmsisi-2d operation idx end --------------*/
    __ARM_2D_IO_NUMBER,
    __ARM_2D_IO_DEFAULT_COPY = __ARM_2D_IO_COPY,
    __ARM_2D_IO_DEFAULT_FILL = __ARM_2D_IO_FILL,
};
//! @}

struct __arm_2d_io_table {
ARM_PRIVATE(
    struct {
        arm_fsm_rt_t    (*SW)(__arm_2d_sub_task_t *ptTask);
        arm_fsm_rt_t    (*HW)(__arm_2d_sub_task_t *ptTask);
    }OP[__ARM_2D_IO_NUMBER ];
)};

...

__WEAK
const struct __arm_2d_io_table __ARM_2D_IO_TABLE = {
    .OP = {
        [__ARM_2D_IO_COPY] = {
            .SW = (__arm_2d_io_func_t *)&__arm_2d_sw_tile_copy,
        },
        [__ARM_2D_IO_FILL] = {
            .SW = (__arm_2d_io_func_t *)&__arm_2d_sw_tile_fill,
        },
        ...
        [__ARM_2D_IO_ALPHA_BLENDING] = {
            .SW = (__arm_2d_io_func_t *)&__arm_2d_sw_alpha_blending,
        },
        ...
        [__ARM_2D_IO_COLOUR_CONVERT_TO_RGB565] = {
            .SW = (__arm_2d_io_func_t *)__arm_2d_sw_convert_colour_to_rgb565,
        },
        ...
        
    },
};

4 Documentation

Name Description Location
README.md It is the document that you are reading. It provides basic information and guidance for the arm-2d library. (root)
how_to_deploy_the_arm_2d_library.md A step by step guide that helps you to deploy the library to your existing or new projects. (root)
introduction.md A relatively detailed introduction for the library, including basic concepts, programmers' mode etc. documents
how_to_use_tile_operations.md A detailed document elaborates the APIs dedicated to basic tile operations in the arm-2d library. documents
how_to_use_alpha_blending_operations.md A detailed document elaborates the APIs dedicated to alpha-blending services provided by the arm-2d library. documents
how_to_use_conversion_operations.md A detailed document elaborates the APIs dedicated to colour space conversion services provided by the arm-2d library. documents
how_to_use_drawing_operations.md A detailed document elaborates the APIs that provide basic point-drawing and colour-filling services in the arm-2d library. documents

5 Limitations

5.1 The Generic Limitations

  • The library focus on Cortex-M processors in principle.
  • The library should be compiled with the following compilers:
    • Arm Compiler 5
    • Arm Compiler 6
    • GCC
    • LLVM
    • IAR
  • The library focus on Low Level Pixel Processing Acceleration
    • In principle, the library will NOT provide APIs for content creation, such as drawing shapes, text display etc., but simple point drawing APIs.
    • In principle, the library will NOT provide data structures or related algorithms essential for creating a GUI, for example, element tree, GUI message handling and the tree traversal algorithms.

5.2 The Temporary Limitations

  • The library currently is developed and validated using Arm Compiler 6.
  • The library currently can only be used in the C environment. C++ support will be added later.
  • The library currently only supports two specific colours, i.e. RGB565 and RGB888
  • Anti-aliasing algorithms haven't been implemented yet.
  • Rotation, Zooming (Stretching) algorithms haven't been implemented yet.
  • Alpha-masking algorithms haven't been implemented yet.
  • The library currently only provides default software algorithms and a Helium based acceleration library.
    • Although planned, no hardware acceleration is implemented or supported for now.
    • Although planned and implemented, the ACI (Arm Custom Instruction) acceleration solutions are not open-source for now. Please contact local Arm FAE for details.
  • The provided example projects only run on MPS2, MPS3, FVP and some 3rd party development platforms, e.g. STM32F746G-Discovery.
    • Feel free to try the library on your own devices. The library depends on No specific peripherals.
  • Example projects are based on MDK (one of the most popular development environments for Cortex-M processors ).

6 Feedback

As mentioned at the beginning, the purpose of this project is to explore and demonstrate the possibilities of delivering smart-phone modern graphic user interface using low-cost and resource constraint micro-controllers. We expect that this arm-2d library could inspire more similar initiatives and engineering practices. Hence, your feedback and thoughts are precious to us. On the other hand, although this is a research project, with your help and support, it can be promoted as a formal product similar to other open-source projects delivered by Arm.

If you want to spend some time to try the library and leave some thoughts, please feel free to raise issues and kindly provide us with some critical information such as:

  • The target application and industry segment which you want to introduce a GUI using Cortex-M processors
  • The constraint of your platform, such as the size of the RAM, ROM, system frequency, the average power consumption etc.
  • The LCD resolution and target frame-rate (FPS)
  • Algorithms that you like and don't like
  • Algorithms that are missing in the library and why you recommend them.
  • What kind of device do you use?
    • Does it contain HW accelerators for 2D image processing?
    • What kind of features/functionalities does the accelerator provide?
    • Does the HW accelerator contain features that arm-2d currently doesn't support?
  • Any other thoughts or suggestions.

Thank you for your time.

Arm-2D Development Team.