Office File API — Enhance Accessibility in Office Documents (Word & Excel) using OpenAI Models

For accessibility-related reasons, certain business usage scenarios require that Office documents include meaningful descriptions/alternative text (Alt Text) for graphics content (images, charts, etc). Alt Text helps individuals with visual impairments to interpret pictures and other graphical content (screen readers cannot properly read documents that contain images without Alt Text). In addition, documents without Alt Text cannot be exported correctly as accessible PDFs (these PDFs fail accessibility validation).

The OpenAI platform offers a solution to address this particular requirement/issue (OpenAI's generative AI can help describe images and other graphical content within Word and Excel files). In this blog post, I'll show you how to integrate the OpenAI model into a DevExpress-powered Office File API application and add missing media descriptions and Alt Text. After processing documents, users can export files to accessible PDFs or upload them in a document viewer and read with a screen reader.

We created a sample project to help illustrate our strategy. This REST API application includes two endpoints: to describe images in Word documents and charts in Excel files.
You can download this project from GitHub: Office File API – Integrate AI to Generate Accessible Descriptions

Implement OpenAI Model API

Before you incorporate this solution in your app, please be sure to read and understand OpenAI's license agreement and terms of use.

To begin, add a reference to the Azure.AI.OpenAI package in your project. This package adapts OpenAI's REST APIs so they can be used in non-Azure OpenAI development.
We will use this API to send requests and process responses.

The following code snippet sends a request to describe an image and obtain a string with a response.

class OpenAIClientImageHelper {
    OpenAIClient client;
    internal OpenAIClientImageHelper(string openAIApiKey) {
        client = new OpenAIClient(openAIApiKey, new OpenAIClientOptions());
    }
    string ConvertDXImageToBase64String(DXImage image) {
        using (MemoryStream stream = new MemoryStream()) {
            image.Save(stream, DXImageFormat.Png);
            byte[] imageBytes = stream.ToArray();
            return Convert.ToBase64String(imageBytes);
        }
    }
    internal async Task<string> DescribeImageAsync(OfficeImage image) {
        string base64Content = ConvertDXImageToBase64String(image.DXImage);
        string imageContentType = OfficeImage.GetContentType(OfficeImageFormat.Png);
        return await GetImageDescription($"data:{imageContentType};base64,{base64Content}");
    }
    internal async Task<string> GetImageDescription(string uriString) {
        ChatCompletionsOptions chatCompletionsOptions = new() {
            DeploymentName = "gpt-4-vision-preview",
            Messages =
                {
                    new ChatRequestSystemMessage("You are a helpful assistant that describes images."),
                    new ChatRequestUserMessage(
                        new ChatMessageTextContentItem("Give a description of this image in no more than 10 words"),
                        new ChatMessageImageContentItem(new Uri(uriString))),
                },
            MaxTokens = 300
        };

        Response<ChatCompletions> chatResponse = await client.GetChatCompletionsAsync(chatCompletionsOptions);
        ChatChoice choice = chatResponse.Value.Choices[0];
        return choice.Message.Content;
    }
}

Word Processing Document API Endpoint

We can use the API described above within a DevExpress-powered Word Processing Document API application. Use the Document.Shapes collection to retrieve document images. Sort the retrieved shapes by type and check whether the image includes Alt Text. Then call the OpenAIClientImageHelper.GetImageDescription method implemented above to generate an image description.

public async Task<IActionResult> GenerateImageAltText(IFormFile documentWithImage,
    [FromQuery] RichEditFormat outputFormat) {
    try {
        var imageHelper = new OpenAIClientImageHelper(openAIApiKey);
        using (var server = new RichEditDocumentServer()) {
            await RichEditHelper.LoadFile(server, documentWithImage);
            server.GenerateAltTextForImages((document) => {
                foreach (var shape in document.Shapes) {
                    if (shape.Type == DevExpress.XtraRichEdit.API.Native.ShapeType.Picture && string.IsNullOrEmpty(shape.AltText))
                        shape.AltText = imageHelper.DescribeImageAsync(shape.PictureFormat.Picture).Result;
                }
            });

            Stream result = RichEditHelper.SaveDocument(server, outputFormat);
            string contentType = RichEditHelper.GetContentType(outputFormat);
            return File(result, contentType, $"result.{outputFormat}");
        }
    }
    catch (Exception e) {
        return StatusCode(500, e.Message + Environment.NewLine + e.StackTrace);
    }
}

Spreadsheet Document API Endpoint

You can also generate Alt Text for Excel Charts. Our project shows how to combine Azure OpenAI and Spreadsheet Document APIs to address this particular requirement.
First, you'll need to convert Excel charts to images. Call the Shape.ExportToImage method to obtain the OfficeImage object from each chart. Then call the OpenAIClientImageHelper.GetImageDescription method implemented above to generate an image description and use it as Alt Text.

public async Task<IActionResult> GenerateChartAltText(IFormFile documentWithImage,
    [FromQuery] SpreadsheetFormat outputFormat) {
    try {
        var imageHelper = new OpenAIClientImageHelper(openAIApiKey);
        using (var workbook = new Workbook()) {
            await SpreadsheetHelper.LoadWorkbook(workbook, documentWithImage);
            foreach (var worksheet in workbook.Worksheets) {
                foreach (var chart in worksheet.Charts) {
                    OfficeImage image = chart.ExportToImage();
                    chart.AlternativeText = await imageHelper.DescribeImageAsync(image);
                }
            }

            Stream result = SpreadsheetHelper.SaveDocument(workbook, outputFormat);
            string contentType = SpreadsheetHelper.GetContentType(outputFormat);
            return File(result, contentType, $"result.{outputFormat}");
        }
    }
    catch (Exception e) {
        return StatusCode(500, e.Message + Environment.NewLine + e.StackTrace);
    }
}

As always, your feedback is very important. Please let us know whether additional AI-related samples/solutions are of interest to you.