강용민

강용민님의 아티클

강용민

강용민

비디오 스트리밍

이번 프로젝트들(온라인 스터디 & 영상 기반 포트폴리오)을 통해 비디오 스트리밍이라는 기술에 손 대볼 기회가 생겼고, 이번 프로젝트들을 하면서 고민했던 문제들에 대해 공유해보려 합니다.


Video Streaming의 종류

비디오 스트리밍 기술이 하나로 이뤄져 있는 줄 알았는데 알아보니 현재 쓰고 있는 기술을 크게 두 가지(WebRTC, HLS)로 나뉘었습니다. 이 두 가지의 기술을 이해하려면 전에 어떤 이유에서 비디오 스트리밍을 사용하는지 다음과 같은 고민이 필요합니다.


  • 영상 흐름 방향은 어떻게 되나??
  • 영상 송신자와 수신자의 비율이 어떻게 되나 고민해야 합니다. 크게 (1: N), (M : N)으로 나눌 수 있습니다.
  • (1 : N) : 송신자 한 명이 여러 수신자에게 영상을 뿌려줍니다. 즉, 단방향 통신입니다. `Youtube나 Instagram 등의 SNS 라이브 방송` 이나 `인터넷 쇼핑`을 예로 들 수 있습니다.
  • (M : N) : 여러 송신자가 여러 수신자에게 영상을 뿌려줍니다. 즉 양방향 통신이 돼야 합니다. 송신자가 수신자가 될 수도 있습니다. `Youtube, Gather와 같은 화상 채팅`을 예로 들 수 있습니다.
  • 지연 시간(Latency)이 있어도 되나??
  • 물론 지연 시간이라는 게 짧으면 짧을수록 좋지만 지연 시간이 좀 있어도 허용 가능한가에 대한 고민해야 합니다. 지연 시간이 적으면 적을수록 그만큼 데이터 통신이 빠르게 오가야 하고, 이는 처리하는 서버에 대한 자원 부담이 됩니다.
  • 지연 시간 허용 가능 : SNS 라이브 방송은 송신자가 영상을 이끌어 가기에 약간의 지연 시간(10~20초)을 허용해줍니다.
  • 지연 시간 허용 불가능 : 회의는 송신자와 수신자가 서로 계속해서 소통을 해야 하기 때문에 지연 시간이 길 수 없습니다.
  • 사용자는 얼마나 될까?
  • 이는 영상 방향과 지연 시간과 연관이 되긴 합니다. 사용자 수가 같다면 모든 사용자의 영상 데이터가 양방향으로 오가야 한다면 그만큼 자원 부담이 커질 테고, 지연 시간도 짧아야 한다면 더욱 커집니다.

이러한 고민을 했다면 이제 기술에 대해 알아보겠습니다.


WebRTC

WebRTC는 웹 브라우저 간 P2P 연결을 통해 데이터 교환을 가능케 하는 시술입니다. 양방향 통신이 가능하며, Latency가 짧은 것이 특징입니다. 하지만 그만큼 자원을 많이 잡아먹는다는 단점이 있어 자세한 정보는 포스팅에 남겨 놓았습니다.


HLS

HLS는 Http Live Streaming의 약자로 HTTP가 들어간 것을 보면 예상할 수 있듯이 HTTP 프로토콜의 일종입니다. WebRTC보다 자원에 대한 부담이 적어 그만큼 많은 사용자를 받아들일 수 있는 것이 특징입니다. 하지만 Latency가 있으며, 양방향 통신에 최적화돼있지 않습니다.


선택

온라인 스터디

온라인 스터디의 경우 서로 소통해야 하고, 방마다 사용자가 많아도 20명 이하일 것이라는 가정하에 WebRTC를 택했습니다. 그중 빠른 프로토타입을 만들고, 비용 절감을 위해 Mesh type 구조를 택했습니다.


영상 기반 포트폴리오

영상 기반 포트폴리오의 경우 SNS 방송처럼 단방향 소통으로 해도 되고, 약간의 지연시간을 허용해도 괜찮기에 HLS 기술을 택했습니다.


이외에도 실시간 비디오 스트리밍이라는 기술을 배우면서 새로 알게 된 지식도 있지만 TCP/UDP처럼 네트워크 개념에 대해 다시 상기할 기회가 됐습니다. 앞으로 Encoding/Decoding, Caching 등 여러 해결해야 할 길이 많지만 하나하나 해결하다 보면 언젠가는 부끄럽지 않은 프로젝트가, 사람이 되기를 희망하며 마칩니다.

4
0
강용민

강용민

Online-Study_Chapter01

This chapter was intended to implment WebRTC, but we decide to learn Websocket first beacuse Websocket used in the signaling process.

I think lif is not going as expected. Anyway, let's try it.


WebSocket

Websocket is a protocol developed to provide full duplex in compatibility with half duplex HTTP protocol.

Unlike general Socket communication, it uses HTTP 80 Port, so there are no restrictions on firewalls and is usually called WebSocket.

HTTP protocol is used for access, and communication is then communicated using its own WebSocket protocol.


Socket vs WebSocket

we have something to watch out for : socket and web socket are different.

First of all, socket and web socket are similar in that they communicate through IP and port. In addition, both have similar characteristics of full duplex. But websocket operate on HTTP layers, so it's different from the layer on the TCP/IP socket!

Operating layer: Based on OSI layer 7, the socket is based on the Internet protocol (TCP/IP layer), so it is located in layer 4 (network layer) to which TCP and UDP belong, and the Websocket is based on HTTP, so it is located in layer 7(application layer).

Data format: Socket communication based on TCP is simply data transmission through a byte stream, so data consisting of bytes must be handled, but Websocket communication is based on layer 7, which is an application layer, so it deals with data in the form of a message.


Websocket Access Process

In order for a server and a client to communicate using a Websocket, they must first go through a Websocket connection process. The Websocket access process can be divided into TCP/IP access and the web socket opening handshake process. Because Websocket also operate on TCP/IP, servers and clients must have TCP/IP connections with each other before using the Websocket. After the TCP/IP connection is completed, the server and the client begin the websocket opening handshake process.


Building a Springboot Websocket server

I think we've learned enough about the Websocket, so let's understand more while creating a chat app.


Add Library

First, add 'spring webSocket' to 'build.gradle' to use webSocket in Spring


build.gradle

plugins {
    id 'java'id 'org.springframework.boot' version '3.0.5'id 'io.spring.dependency-management' version '1.1.0'
}

//...

dependencies {
    implementation 'org.springframework.boot:spring-boot-starter-data-jpa'
    implementation 'org.springframework.boot:spring-boot-starter-hateoas'
    implementation 'org.springframework.boot:spring-boot-starter-web'
    
    // Add Websocket Library
    implementation 'org.springframework.boot:spring-boot-starter-websocket'

    implementation 'org.springframework.boot:spring-boot-starter-validation'
    testImplementation 'org.projectlombok:lombok:1.18.22'
    testImplementation 'org.springframework.boot:spring-boot-starter-validation'
    compileOnly 'org.projectlombok:lombok'
    runtimeOnly 'com.h2database:h2'
    runtimeOnly 'com.mysql:mysql-connector-j'
    annotationProcessor 'org.projectlombok:lombok'
    testImplementation 'org.springframework.boot:spring-boot-starter-test'
}


ChatDTO : Chat Message Info

I need a DTO to send and receive chat messages. Depending on the situation, there are two situations: entering the chat room and sending a message to the chat room, so ENTER (entering the chat room) and TALK (talking) are declared enum. The remaining member fields consist of chat room identification ID, message sender, and message.


ChatDTO.java

package com.example.webrtc.domain.webChat.dto;

import lombok.Builder;
import lombok.Data;

@Data
@Builder
public class ChatDTO {
    public enum MessageType{
        Enter, TALK
    }

    private MessageType type;
    private String roomId;
    private String sender;
    private String message;
    private String time;
}


Websocket Handler

socket communication involves a 1:N relationship between the server and the client. That is, multiple clients may access one server. Therefore, the server needs a handler to handle messages sent by multiple clients. We will implement text-based chat, so we will inherit 'TextWebSocketHandler' and write it. It outputs the message received from the client to the Console-log and sends a welcome message to the client.


WebSocketHandler.java

package com.example.webrtc.global.config;

//...

@Slf4j
@Component
@RequiredArgsConstructor
public class WebSocketHandler extends TextWebSocketHandler {
    private final ObjectMapper mapper;
    private final ChatService chatService;


    protected void handleTextMessage(WebSocketSession session, TextMessage message) throws IOException {
        String payload = message.getPayload();
        log.info("payload {}", payload);

        ChatDTO chatMessage = mapper.readValue(payload, ChatDTO.class);
        log.info("session {}", chatMessage.toString());

        ChatRoom room = chatService.findRoomById(chatMessage.getRoomId());
        log.info("room {}", room.toString());

        room.handleAction(session, chatMessage, chatService);
    }
}


What is Payload?

Payload means the data being transmitted. When transmitting data, various elements such as header, META data, and error check bits are sent together to increase data transmission efficiency and stability. At this time, the payload means the data you want to send itself. For example, when you send and receive a parcel delivery, it is not a payload because the package is a payload and the invoice or box is an additional one.


Websocket Config

Create a Config to enable WebSocket using a handler. Use the '@EnableWebSocket' annotation to activate WebSocket. The endpoint for accessing WebSocket is set to 'ws/chat'. Add CORS: setAllowedOprigins(" * "); so that the domain can also be accessed from other servers.


WebSocketConfig.java

package com.example.webrtc.global.config;

//...

@Configuration
@EnableWebSocket
@RequiredArgsConstructor
public class WebSocketConfig implements WebSocketConfigurer {

    private final WebSocketHandler webSocketHandler;

    public void registerWebSocketHandlers(WebSocketHandlerRegistry registry) {
        registry.addHandler(webSocketHandler, "/ws/chat")
                .setAllowedOrigins("*");
    }
}


ChatRoom

The chat room must have information about the clients in that chat room. The chat room must have a client-specific session for the client who entered the chat room. To do this, create a HashSet named sessions to store sessions on a per-client basis. Next, take the ID of the chat room and the name of the chat room as variables.

The method declares a total of two things.

  • handleAction : A method for forwarding a message to a session (client) according to the Message Type. If the type is 'ENTER', display "Welcome" in the chat room, and if it is 'TALK', reflect the message that the client has occurred in the chat room as it is.
  • sendMessage : A method that allows messages from handAction to be delivered to all sessions contained in sessions.


ChatRoom.java

package com.example.webrtc.domain.webChat.dto;

//...

@Data
public class ChatRoom {
    private String roomId;
    private String name;
    private Set<WebSocketSession> sessions = new HashSet<>();

    @Builderpublic ChatRoom(String roomId,String name){
        this.roomId = roomId;
        this.name = name;
    }

    public void handleAction(WebSocketSession session, ChatDTO message, ChatService chatService){
        if(message.getType().equals(ChatDTO.MessageType.Enter)){
            sessions.add(session);

            message.setMessage(message.getSender() + " 님이 입장하셨습니다.");
            sendMessage(message,chatService);
        }
    }

    private <T> void sendMessage(T message, ChatService chatService) {
        sessions.parallelStream().forEach(session -> chatService.sendMessage(session,message));
    }
}

As I write, I realize that Handler is a typical example of an observer pattern. Why didn't I know all this time?


ChatService

'createRoom()', 'findRoomById()' used here should be transferred to DAO as soon as they are actually connected to DB. We're still planning to make it without connecting to DB, so I put it in the service first. Because there is no connection with DB, the chat room information is saved in HashMap.

  • CreateRoom : Set the chat room ID with the UUID value randomly generated through UUID, and create a chat room by setting the chat room name with NAME.
  • sendMessage : Sends a message to the specified session.


ChatService.java

package com.example.webrtc.domain.webChat.service;

//...

@Slf4j
@Data
@Service
public class ChatService {
    private final ObjectMapper mapper;
    private Map<String, ChatRoom> chatRooms;

    @PostConstructprivate void init(){
        chatRooms = new LinkedHashMap<>();
    }

    public ChatRoom findRoomById(String roomId){
        return chatRooms.get(roomId);
    }

    public ChatRoom createRoom(String name){
        String roomId = UUID.randomUUID().toString();

        ChatRoom room = ChatRoom.builder()
                .roomId(roomId)
                .name(name)
                .build();

        chatRooms.put(roomId,room);
        return room;
    }

    public <T> void sendMessage(WebSocketSession session, T message){
        try{
            session.sendMessage(new TextMessage(mapper.writeValueAsString(message)));
        } catch (IOException e) {
            log.error(e.getMessage(),e);
        }
    }
}


ChatController

The creation and inquiry of the chat room will be implemented with REST api, so create a controller.


ChatController.java

package com.example.webrtc.domain.webChat.Controller;

//..

@RestController
@Slf4j
@RequiredArgsConstructor
@RequestMapping("/chat")
public class ChatController {
    private final ChatService chatService;

    @PostMappingpublic ChatRoom createRoom(@RequestParam String name){
        return chatService.createRoom(name);
    }
}


Finish

Done. Now let's run the application to create a room and chat based on roomId.



[References]

https://www.daddyprogrammer.org/post/4077/spring-websocket-chatting/

https://supawer0728.github.io/2018/03/30/spring-websocket/

https://velog.io/@rhdmstj17

3
0
강용민

강용민

Online-Study_Chapter00

Summary

This project will develop a video chat-related service that has been delayed under the pretext of "I think it will be difficult" and "I don't have time". Real-time remids me of the problem of delay and connections even if I don't actually lear it. I don't know if the project will success, but...let's try it.

 

In order to implement video chat, I need to know a technology called Web Real-Time-Communication(Web RTC), so I will study this first.

Of course, WebRTC is not the only technology in live video streaming. Depending on the delay in live video streaming technology, there are many technologies such as the following.

Let's look at RTMP, WebRTC, And HLS, the three most commonly used video protocols.

  • RTMP(Real-Time Messaging Protocol)
  • The primary intent of this protocol was to achieve low latency and reliable communication by maintaining a persistent connection between the streaming server and the video player. In layman’s terms, it can deliver a reliable video stream from the source to the viewer in under 5 seconds of delay on average.
  • As more modern clients like mobile and IoT emerged in the last decade, RTMP started losing its ground due to its inability to support native playback in these platforms. 
  • HLS(HTTP Live Streaming)
  • Following the rise of HTML5, HLS took over the reins of video streaming by offering robust, reliable media distribution that can scale to hundreds of millions of users with the support of CDNs.Today, HLS is widely used by all browsers, mobile, and LR platforms to facilitate video consumption and streaming. HLS also supports DRM encryption (Digital Rights Management), a high security mechanism to protect the content it is streaming. This makes it the ideal choice for last-mile content delivery by all streaming and OTT providers
  • For live video streaming, the main drawback of HLS is latency.Taking into account the time for ingesting the source, generating multiple renditions, distribution over CDNs, and final playback buffer requirements, latency can easily peak at 20-30 seconds in HLS.
  • WebRTC
  • WebRTC works on a long stack of protocols to abstract the media engine, codec, and transport layer into a bunch of APIs. The web developer just has to use these APIs to capture the media from a webcam, set up a peer connection, and transmit the data directly between any two browsers across the world with less than 500 milliseconds of latency.
  • The challenge, however, lies in scaling. Due to the intense bandwidth configuration required to support multiple peer connections, WebRTC natively does not scale well beyond a few thousand connections at most.


Considering these characteristics and the caracateristics of the study(used by fewer than 10 people at most), I chose WebRTC because it was suitable for the project.

 

What is WebRTC?

as its name suggests, WebRTC can add real-time communicatioin function to applications. The WebRTC website can see the phrase "It supports video, voice, and generic data to be sent between peers, allowing developers to build powerful voice- and video-communication solutions." for the technology. Then let's learn about the technologies and protocols that WebRTC included.

 

ICE(Interactive Connectivity Establishment)

ICE is framework used in computer networking to find ways for two computers to talk to each other as directly as possible in peer-to-peer networking, but presence of NATs and Firewalls make it difficult for nodes(computers) to communicate with each other. It is a Networking technique which makes use of STUN(Session Traversal Utilities for NAT) and TURN(Traversal Using Relays Around NAT) to establish a connections between to nodes which is as direct as posiible.

To understand how ICE works, we need to know the workings of STUN and its extension TURN.

 

STUN(Session Traversal Utilities for NAT)

STUN and TURN servers are "servers that tell our where to communicate with each other and where to communicate". If we think about it, when someone asks where we commincate with each other on the Internet, we can answer that we communicate through IP. After all, STUN and TURN servers are servers thath inform each other's IPs.

For an endpoint under a NAT which has a local address, it is not reachable for other endpoints outside the local network, Hence a connection cannot be established. When this occurs the endpoint can request it’s public IP address from a STUN server. This publicly reachable IP can be used by other endpoints to establish a connection. But this case fails when endpoints are under symmetric NAT, which happens in most of the practical cases.This is where a TURN server comes into picture.

 

TURN(Traversal Using Relays Around NATs)

TURN server as the name suggests is used as a relay server or an intermediate server to exchange data. When any endpoint under Symmetric NAT can contact a TURN server which is on the public internet to establish a connection the endpoint is then called a TURN client. The disadvantage of using a TURN server is that it is required throughout the whole time span of the session unlike STUN server which no longer needed after the connection is established. Therefore in ICE technique STUN is used as default.

 

SDP(Session Description Protocol)

Through ICE, we identified the address candidates capable of P2P commuications.Now, we have to make information exchange with each other, which is SDP.

Session Description Protocol (SDP) is a protocol used in WebRTC to describe the initial acquisition of multimedia content, such as the resolution, format, and codec of streaming media. Video resolution, audio transmission or reception may be transmitted and received. hmm...this part is hard to understand.

 

Type of WebRTC implementations

we learn about the types of WebRTC implementations.

There are three main implementation methods of WebRTC : Mesh, MCU, and SFU.

 

Mesh

It's P2P method that we can easily think of. As shown in the figure above, each client has one Uplink to transmit its media information and one Downlink to receive media information. the server is used only to relay information for the connections because the connection between peers proceeds.

If five clients form Peer Connection for each as shown in the figure above, each client will have a total of eight links, with four uplinks to send their media information to the remaining four clients and four downlinks to receive media information from four clients. the load on the server is small, but the more clients there are, the more the load on the client becomes.

 

MCU(Multi-point Control Unit)

Unlike P2P, the server relays media traffic.

The MCU mixes, processes, and transmits media information from each client to the receiving side. Therefore, it is not necessary to have N downlinks to receive media information, but only one downlink to receive mixed data. In other words, it is an optimized method for the network because only two links, one Uplink to transmit media information and one Downlink to receive media information, need to be maintained.

However, as mentioned above, the load on the server increases exponentially because media information from all connected clients must be received, mixed, and processed.

 

SFU(Selective Forwarding Unit)

like MCU, the server relay media traffic too. Hower, the difference from the MCU is that the server does not mix and process media infromation from the client side, but forwards it as it is. In a way, it can be seen as an intermediate method between Mesh and MCU

As shown in the figure above, we will have one Uplink to transmit its media information and N Downlinks to receive media information. In other words, both the client and the server share the load

 

Our video study has a zoom-like format. In an environment where real-time guaranteed, several people need to access and exchange video and voice data, but the number of participants may be small or large. In addition, we excluded P2P that look bad in N:M.

And we tried to use SFU as a server cost problem between SFU and MCU.

6
2