Introduction

Greetings on the 14th day of the REALITY Advent Calendar 2023! Today, we're thrilled to talk about an experimental feature called Stream Preview, developed during the REALITY Engineer Study Camp 2023. 

Before we delve into the details, a special acknowledgment to あさだ(Asada), the writer of the 13th day publication, "REALITYにショート動画の投稿機能を入れてみた (Part 2)/ Short video posting function for REALITY (Part 2)" . If you haven't caught up on that, it's worth a read to stay in the loop.

Meet the Writers: Aru & Yazaki

The minds behind today's article is the dynamic duo— Aru, the server sorcerer, and Yazaki, the maestro of iOS magic!

Background

In the dynamic world of streaming, where motion conveys more than static images, we sought to elevate user experience through Stream Preview—a feature designed give viewers a sneak peek into streams directly from the home screen.

Requirements/Objectives

Following were the key requirements that we aimed to develop in the study camp:

• Allow streamers to record a short clip during a live stream.

• Upload the clip.

• Use the uploaded clip as a thumbnail for streams that show up in the viewer's recommended tab.

While these requirements are not aimed for production release, they will help us better understand how such a feature could be used and the potential challenges.

Server Implementation

On the server side, we implemented an API endpoint that the client app can use to upload a preview clip/thumbnail for a given stream. We use Protocol Buffers for client-server communication. The .proto file defining the request and response for the aforementioned endpoint is as follows:

syntax = "proto3";
package hoge;

/**
 * <strong>POST /api/preview/post</strong>
 * Retrieve URLs for uploading the stream-preview movie and its thumbnail
*/
message StreamPreviewPostRequest {
    int64 streamId = 1;
}

message StreamPreviewPostResponse {
    string movieBodyUrl = 1;
    string movieThumbnailBodyUrl = 2;
}

In the request body, the client includes the ID of a given stream. The server responds with two URLs that the app can use to upload files using PUT method, one for the preview clip and one for the thumbnail of the clip. The URLs are GCS bucket paths where the files are eventually going to be stored.

func GetSignedUrlForPreview(r *http.Request, c context.Context, config *AppConfig, req *StreamRequest) (string, error) {

    file, err := BuildFileName(config.StreamPreviewSettings.BasePath, req.StreamId, "movie")
    if err != nil {
        return "", err
    }

    // set to expire in 15mins
    url, err := GetSignedURL(r, c, config.StreamSettings.BucketName, file, 15*time.Minute)
    if err != nil {
        return "", fmt.Errorf("error generating movie URL: %w", err)
    }

    return url, nil
}

func BuildFileName(basePath, streamID, fileType string) (string, error) {
    timestamp, err := time.Now().UTC().Format(time.RFC3339Nano)
    if err != nil {
        return "", fmt.Errorf("error formatting timestamp: %w", err)
    }

    return fmt.Sprintf("%s/%s/%s/%s", basePath, streamID, timestamp, fileType), nil
}

func GetSignedURL(req *http.Request, context context.Context, bucketName, objectName string, expiration time.Duration) (string, error) {
    client, err := storage.NewClient(context)
    if err != nil {
        return "", fmt.Errorf("error creating storage client: %w", err)
    }

    signedURL, err := client.SignedURL(bucketName, objectName, storage.HTTPMethodGet, expiration)
    if err != nil {
        return "", fmt.Errorf("error generating signed URL: %w", err)
    }

    return signedURL, nil
}

The server also stores the GET method URLs of the uploaded files as a Redis hash value using the stream ID as the key. These will be later sent in the response of stream list API, so that the client can use them to display the preview. Since this data will be "read heavy," we chose Redis instead of a database. Besides, the data is relevant as long as the streams are ongoing; therefore, we do not need to persist them long term.

Capturing Preview Clip (Client)

At first, we need to let the user upload a clip. REALITY app has the ability to record a short video, so we use that feature and upload the video to the server instead of saving it locally. In UI, it will look like this:

We also capture the first frame of the clip to be used as thumbnail. The clip and the thumbnail are both uploaded to the server for later use.

Playing Preview Clip (Client)

For this part, we created a LivePreviewCell component that will be rendered for each stream in the recommended tab.

class LivePreviewCell: UICollectionViewCell, NibReusable {
    static let size: CGSize = {
        let screenWidth = UIScreen.main.bounds.width
        // (画面幅 - (左のマージン16 + セル間の余白8 + 右の余白8 + ３つ目の枠がチラ見えする幅8)) / セル2個
        let width = (screenWidth - (16.0 + 8.0 + 8.0 + 8.0)) / 2.0
        let height = width * (15.0 / 9.0)
        return CGSize(width: width, height: height)
    }()

    @IBOutlet private var playerView: AVPlayerLayerView!
    @IBOutlet private var thumbnailView: UIImageView!

In the LivePreviewCell class, the playerView plays the preview clip. The preview, however, will be played when the cell comes into "focus" (more on it later). Other times, thumbnailView will be displayed by default. The size property of the LivePreviewCell class defines a static CGSize for the cell.

Note : We used Youtube app's shorts thumbnail as a reference, which happens to be 15:9 (not 16:9)

In recommended tab of REALITY, picked-up streams are displayed in a horizontally scrollable component. To ensure all the streams gets the viewers attention, we play the previews in turn(one at a time). Once currently played preview is finished, we move to the next one

 case .livePreview(let media, let player):
     guard let self, let cell = cell as? LivePreviewCell else { return cell }
         cell.setStream(media: media, player: player, shouldPlay: viewModel.currentPlayIndex.map { $0 == indexPath.row }) { [weak viewModel] in
             viewModel?.didPlayToEnd(index: indexPath.row)
         }

we also make sure the currently playing preview is in focus of the viewer, so we center the corresponding component

viewModel.currentPlayIndex
    .drive { [weak self] index in
        self?.collectionView.scrollToItem(at: IndexPath(row: index, section: 0), at: .centeredHorizontally, animated: true)
    }
    .disposed(by: disposeBag)

Finally, we have our stream preview!

Challenges

As the streamer needs to manually upload previews during a live stream, for some users, it might be a bit inconvenient. Therefore, the UX needs to be designed with the convenience of the streamer in mind. Also, since the previews are only "close to real-time" (depends on upload frequency), strategies need to be in place to ensure consistent preview uploads at all times.

Conclusion

In conclusion, the journey of implementing live stream previews was challenging, but the experiment was undoubtedly worthwhile. Stream previews have the power to captivate viewers, arouse curiosity, and drive increased user engagement.

A special shout out to かとう(Kato), the writer of tomorrow's publication, "カラオケできるギフトを作ったよ！/ I made a karaoke singing gift!" . Join us again tomorrow for the 15th day of REALITY Advent Calendar 2023!