流媒体学习之路(WebRTC)——FEC逻辑分析（6）
——
我正在的github给大家开发一个用于做实验的项目 —— github.***/qw225967/Bifrost
目标：可以让大家熟悉各类Qos能力、带宽估计能力，提供每个环节关键参数调节接口并实现一个json全配置，提供全面的可视化算法观察能力。
欢迎大家使用
——
在讲具体内容之前插一句嘴，从G***分析（3）开始，我们将针对G***的实现细节去分析它设计的原理，让我们理解这些类存在的意义，不再带大家去串具体的流程了。
一、Fe***ontrollerDefault
1.1 背景介绍
前面为大家介绍了G***的部分逻辑，G***作为码率控制的第一环所有的数据都会收到它输出的参考码率的限制。在发送的数据内容中有很多的分类：视频数据、音频数据、重传数据、FEC数据。FEC前面有向大家介绍过，中文称为：前向纠错。在WebRTC中有一个很好的Fec应用例子，它通过RR回复的网络信息进行动态调整，使得在传输中抵抗不同程度的网络损伤。下面我针对FEC给大家介绍一下动态变化的逻辑。
Fe***ontroller是WebRTC提供的接口，如果需要自定义自己的控制类，那么继承它之后进行开发就可以了。
class Fe***ontroller {
public:
virtual ~Fe***ontroller() {}
virtual void SetProtectionCallback(
VCMProtectionCallback* protection_callback) = 0;
virtual void SetProtectionMethod(bool enable_fec, bool enable_nack) = 0;
// Informs loss protectoin logic of initial encoding state.
virtual void SetEncodingData(size_t width,
size_t height,
size_t num_temporal_layers,
size_t max_payload_size) = 0;
// Returns target rate for the encoder given the channel parameters.
// Inputs: estimated_bitrate_bps - the estimated ***work bitrate in bits/s.
// actual_framerate - encoder frame rate.
// fraction_lost - packet loss rate in % in the ***work.
// loss_mask_vector - packet loss mask since last time this method
// was called. round_trip_time_ms - round trip time in milliseconds.
virtual uint32_t UpdateFecRates(uint32_t estimated_bitrate_bps,
int actual_framerate,
uint8_t fraction_lost,
std::vector<bool> loss_mask_vector,
int64_t round_trip_time_ms) = 0;
// Informs of encoded output.
virtual void UpdateWithEncodedData(
size_t encoded_image_length,
VideoFrameType encoded_image_frametype) = 0;
// Returns whether this FEC Controller needs Loss Vector Mask as input.
virtual bool UseLossVectorMask() = 0;
};
在WebRTC中提供了一个默认的Fec控制类：Fe***ontrollerDefault。
class Fe***ontrollerDefault : public Fe***ontroller {
public:
Fe***ontrollerDefault(Clock* clock,
VCMProtectionCallback* protection_callback);
explicit Fe***ontrollerDefault(Clock* clock);
~Fe***ontrollerDefault() override;
Fe***ontrollerDefault(const Fe***ontrollerDefault&) = delete;
Fe***ontrollerDefault& operator=(const Fe***ontrollerDefault&) = delete;
void SetProtectionCallback(
VCMProtectionCallback* protection_callback) override;
void SetProtectionMethod(bool enable_fec, bool enable_nack) override;
void SetEncodingData(size_t width,
size_t height,
size_t num_temporal_layers,
size_t max_payload_size) override;
uint32_t UpdateFecRates(uint32_t estimated_bitrate_bps,
int actual_framerate_fps,
uint8_t fraction_lost,
std::vector<bool> loss_mask_vector,
int64_t round_trip_time_ms) override;
void UpdateWithEncodedData(size_t encoded_image_length,
VideoFrameType encoded_image_frametype) override;
bool UseLossVectorMask() override;
float GetProtectionOverheadRateThreshold();
private:
enum { kBitrateAverageWinMs = 1000 };
Clock* const clock_;
VCMProtectionCallback* protection_callback_;
Mutex mutex_;
std::unique_ptr<media_optimization::VCMLossProtectionLogic> loss_prot_logic_
RTC_GUARDED_BY(mutex_);
size_t max_payload_size_ RTC_GUARDED_BY(mutex_);
const float overhead_threshold_;
};
Fec的控制类会根据丢包和RTT进行保护因子的计算，最终获得一个动态的保护因子。
丢包和RTT的数据需要根据对端回复的RR而进行动态调整，因此可以理解为：Fec动态调整的粒度依赖于RR回复的频率。我们可以考虑调整RR的反馈频率，来获得自己理想的动态调整粒度，做进一步的优化。
下面我展示一部分cpp的代码（modules/video_coding/media_opt_util.***），进行解析。
1.2 VCMLossProtectionLogic
// 可以设置不同的保护方法，Nack/Fec/Nack+Fec，在下一节分析一下几个方法
void VCMLossProtectionLogic::SetMethod(
enum VCMProtectionMethodEnum newMethodType) {
if (_selectedMethod && _selectedMethod->Type() == newMethodType)
return;
switch (newMethodType) {
case kNack:
_selectedMethod.reset(new VCMNackMethod());
break;
case kFec:
_selectedMethod.reset(new VCMFecMethod());
break;
case kNackFec:
_selectedMethod.reset(new VCMNackFecMethod(kLowRttNackMs, -1));
break;
case kNone:
_selectedMethod.reset();
break;
}
UpdateMethod();
}
...
// 过滤丢包数据，可以 平均过滤/最大值过滤/不过滤
uint8_t VCMLossProtectionLogic::FilteredLoss(int64_t nowMs,
FilterPacketLossMode filter_mode,
uint8_t lossPr255) {
// Update the max window filter.
UpdateMaxLossHistory(lossPr255, nowMs);
// Update the recursive average filter.
_lossPr255.Apply(rtc::saturated_cast<float>(nowMs - _lastPrUpdateT),
rtc::saturated_cast<float>(lossPr255));
_lastPrUpdateT = nowMs;
// Filtered loss: default is received loss (no filtering).
uint8_t filtered_loss = lossPr255;
switch (filter_mode) {
case kNoFilter:
break;
case kAvgFilter:
filtered_loss = rtc::saturated_cast<uint8_t>(_lossPr255.filtered() + 0.5);
break;
case kMaxFilter:
filtered_loss = MaxFilteredLossPr(nowMs);
break;
}
return filtered_loss;
}
...
1.3 VCMProtectionMethod
下面提到了保护因子计算的逻辑，该逻辑是通过查询 kFecRateTable 表，这个表分了很多个部分。每部分的数据都是从 0 ~ X作为一个部分。因为原先这个表是二维的表：kFecRateTable[rate][loss]，在数学上WebRTC团队把它简化成了一维表：kFecRateTable[k]，其实逻辑差不多，大家可以换算一下：k = rate_i*129 + loss_j。
// VCMNackMethod 比较简单，这里不展开。
// VCMFecMethod 这个类就有部分计算的逻辑：
// 该函数调用的 parameters 参数是从上述的loss中的丢包、RTT、帧率等更新进去的
// _protectionFactorK 关键帧保护因子
// _protectionFactorD p帧保护因子
bool VCMFecMethod::ProtectionFactor(const VCMProtectionParameters* parameters) {
// FEC PROTECTION SETTINGS: varies with packet loss and bitrate
// No protection if (filtered) packetLoss is 0
// 没有丢包直接返回
uint8_t packetLoss = rtc::saturated_cast<uint8_t>(255 * parameters->lossPr);
if (packetLoss == 0) {
_protectionFactorK = 0;
_protectionFactorD = 0;
return true;
}
// Parameters for FEC setting:
// first partition size, thresholds, table pars, spatial resoln fac.
// First partition protection: ~ 20%
// firstPartitionProt 换算出来是 51，该值应用在表中。
// kFecRateTable 是一个每帧包数和丢包率对应的一个表，根据两个参数换算出一个fec_rate。
// 这个表的第一部分就是 0 - 51，因此这个参数代表的是第一部分的最大fec_rate
uint8_t firstPartitionProt = rtc::saturated_cast<uint8_t>(255 * 0.20);
// Minimum protection level needed to generate one FEC packet for one
// source packet/frame (in RTP sender)
// 在 ForwardErrorCorrection 类中 NumFecPackets 计算了至少生成一个Fec的码率，
// 假设 媒体数据为 1 个，那么经过换算之后必须得把 fec_rate 设置为 85 才能生成一个Fec包。
uint8_t minProtLevelFec = 85;
// Threshold on packetLoss and bitRrate/frameRate (=average #packets),
// above which we allocate protection to cover at least first partition.
// 每帧包数量 和 丢包率
// 用于计算查表，表的k为：k = rate_i*129 + loss_j
uint8_t lossThr = 0;
uint8_t packetNumThr = 1;
// Parameters for range of rate index of table.
const uint8_t ratePar1 = 5;
const uint8_t ratePar2 = 49;
// Spatial resolution size, relative to a reference size.
// 计算：rate_i*129
// 换算表的参数根据 宽高 704 * 576 为换算基础值。约大我们需要的增幅值越大。
float spatialSizeToRef = rtc::saturated_cast<float>(parameters->codecWidth *
parameters->codecHeight) /
(rtc::saturated_cast<float>(704 * 576));
// resolnFac: This parameter will generally increase/decrease the FEC rate
// (for fixed bitRate and packetLoss) based on system size.
// Use a smaller exponent (< 1) to control/soften system size effect.
// powf 是平方函数，spatialSizeToRef 的 0.3 平方可以获得一个缩小增长趋势的值，保证了平滑性
const float resolnFac = 1.0 / powf(spatialSizeToRef, 0.3f);
// 计算帧的包数量
const int bitRatePerFrame = BitsPerFrame(parameters);
// Average number of packets per frame (source and fec):
// 计算平均包数
const uint8_t avgTotPackets = rtc::saturated_cast<uint8_t>(
1.5f + rtc::saturated_cast<float>(bitRatePerFrame) * 1000.0f /
rtc::saturated_cast<float>(8.0 * _maxPayloadSize));
// FEC rate parameters: for P and I frame
uint8_t codeRateDelta = 0;
uint8_t codeRateKey = 0;
// Get index for table: the FEC protection depends on an effective rate.
// The range on the rate index corresponds to rates (bps)
// from ~200k to ~8000k, for 30fps
// 包数量根据前面分辨率换算的参数，计算得到 码率/帧率 的一个参数，用来查表
const uint16_t effRateFecTable =
rtc::saturated_cast<uint16_t>(resolnFac * bitRatePerFrame);
uint8_t rateIndexTable = rtc::saturated_cast<uint8_t>(
VCM_MAX(VCM_MIN((effRateFecTable - ratePar1) / ratePar1, ratePar2), 0));
// Restrict packet loss range to 50:
// current tables defined only up to 50%
// 计算：loss_j
// 当前默认只支持到50%的丢包
if (packetLoss >= kPacketLossMax) {
packetLoss = kPacketLossMax - 1;
}
// 最终：k = rate_i*129 + loss_j
uint16_t indexTable = rateIndexTable * kPacketLossMax + packetLoss;
// Check on table index
RTC_DCHECK_LT(indexTable, kFecRateTableSize);
// Protection factor for P frame
codeRateDelta = kFecRateTable[indexTable];
// 查表完成后取了前面的边界条件。
if (packetLoss > lossThr && avgTotPackets > packetNumThr) {
// Set a minimum based on first partition size.
if (codeRateDelta < firstPartitionProt) {
codeRateDelta = firstPartitionProt;
}
}
// Check limit on amount of protection for P frame; 50% is max.
if (codeRateDelta >= kPacketLossMax) {
codeRateDelta = kPacketLossMax - 1;
}
// For Key frame:
// Effectively at a higher rate, so we scale/boost the rate
// The boost factor may depend on several factors: ratio of packet
// number of I to P frames, how much protection placed on P frames, etc.
// 关键帧 和 p帧 的保护需要区分开，一是关键帧更重要、二是关键帧的数据量更多，因此保护的内容需要给更多倾斜
const uint8_t packetFrameDelta =
rtc::saturated_cast<uint8_t>(0.5 + parameters->packetsPerFrame);
const uint8_t packetFrameKey =
rtc::saturated_cast<uint8_t>(0.5 + parameters->packetsPerFrameKey);
const uint8_t boostKey = BoostCodeRateKey(packetFrameDelta, packetFrameKey);
rateIndexTable = rtc::saturated_cast<uint8_t>(VCM_MAX(
VCM_MIN(1 + (boostKey * effRateFecTable - ratePar1) / ratePar1, ratePar2),
0));
uint16_t indexTableKey = rateIndexTable * kPacketLossMax + packetLoss;
indexTableKey = VCM_MIN(indexTableKey, kFecRateTableSize);
// Check on table index
assert(indexTableKey < kFecRateTableSize);
// Protection factor for I frame
codeRateKey = kFecRateTable[indexTableKey];
// Boosting for Key frame.
int boostKeyProt = _scaleProtKey * codeRateDelta;
if (boostKeyProt >= kPacketLossMax) {
boostKeyProt = kPacketLossMax - 1;
}
// Make sure I frame protection is at least larger than P frame protection,
// and at least as high as filtered packet loss.
codeRateKey = rtc::saturated_cast<uint8_t>(
VCM_MAX(packetLoss, VCM_MAX(boostKeyProt, codeRateKey)));
// Check limit on amount of protection for I frame: 50% is max.
if (codeRateKey >= kPacketLossMax) {
codeRateKey = kPacketLossMax - 1;
}
_protectionFactorK = codeRateKey;
_protectionFactorD = codeRateDelta;
// Generally there is a rate mis-match between the FEC cost estimated
// in mediaOpt and the actual FEC cost sent out in RTP module.
// This is more significant at low rates (small # of source packets), where
// the granularity of the FEC decreases. In this case, non-zero protection
// in mediaOpt may generate 0 FEC packets in RTP sender (since actual #FEC
// is based on rounding off protectionFactor on actual source packet number).
// The correction factor (_corrFe***ost) attempts to corrects this, at least
// for cases of low rates (small #packets) and low protection levels.
float numPacketsFl =
1.0f + (rtc::saturated_cast<float>(bitRatePerFrame) * 1000.0 /
rtc::saturated_cast<float>(8.0 * _maxPayloadSize) +
0.5);
const float estNumFecGen =
0.5f +
rtc::saturated_cast<float>(_protectionFactorD * numPacketsFl / 255.0f);
// We reduce cost factor (which will reduce overhead for FEC and
// hybrid method) and not the protectionFactor.
_corrFe***ost = 1.0f;
if (estNumFecGen < 1.1f && _protectionFactorD < minProtLevelFec) {
_corrFe***ost = 0.5f;
}
if (estNumFecGen < 0.9f && _protectionFactorD < minProtLevelFec) {
_corrFe***ost = 0.0f;
}
// DONE WITH FEC PROTECTION SETTINGS
return true;
}
二、FlexfecSender
FlexfecSender是目前默认使用的类，其实底层还在使用UlpfecGenerator来进行fec的封包。每次RR回复后都需要调用SetFecParameters把上面计算出来Fec信息设置到其中。在发送数据时会编译调用AddRtpPacketAndGenerateFec函数来产生Fec数据。然后发送数据前Fec数据和Nack数据都会塞入pacer中，而Fec数据的优先级比Nack低。
// FlexfecSender中的两个函数：SetFecParameters、AddRtpPacketAndGenerateFec。
// 在地下都是 ulpfec_generator_ 直接调用对应的，SetFecParameters、AddRtpPacketAndGenerateFec。
constexpr size_t kUlpfecMaxMediaPackets = 48;
constexpr uint8_t kHighProtectionThreshold = 80;
constexpr size_t kMinMediaPackets = 4;
...
void UlpfecGenerator::SetFecParameters(const FecProtectionParams& params) {
RTC_DCHECK_GE(params.fec_rate, 0);
RTC_DCHECK_LE(params.fec_rate, 255);
// Store the new params and apply them for the next set of FEC packets being
// produced.
new_params_ = params;
// 当fec_rate大于阈值，那么媒体包数最低也要 4 个
if (params.fec_rate > kHighProtectionThreshold) {
min_num_media_packets_ = kMinMediaPackets;
} else {
min_num_media_packets_ = 1;
}
}
...
int UlpfecGenerator::AddRtpPacketAndGenerateFec(const uint8_t* data_buffer,
size_t payload_length,
size_t rtp_header_length) {
RTC_DCHECK(generated_fec_packets_.empty());
if (media_packets_.empty()) {
params_ = new_params_;
}
bool ***plete_frame = false;
// marker_bit 代表了帧的结尾
const bool marker_bit = (data_buffer[1] & kRtpMarkerBitMask) ? true : false;
// 每次最大只能保护 80 个包
if (media_packets_.size() < kUlpfecMaxMediaPackets) {
// Our packet masks can only protect up to |kUlpfecMaxMediaPackets| packets.
std::unique_ptr<ForwardErrorCorrection::Packet> packet(
new ForwardErrorCorrection::Packet());
packet->length = payload_length + rtp_header_length;
memcpy(packet->data, data_buffer, packet->length);
media_packets_.push_back(std::move(packet));
// Keep track of the RTP header length, so we can copy the RTP header
// from |packet| to newly generated ULPFEC+RED packets.
RTC_DCHECK_GE(rtp_header_length, kRtpHeaderSize);
last_media_packet_rtp_header_length_ = rtp_header_length;
}
// 当帧完整了就进行Fec保护，每次保护都是按帧级别进行
if (marker_bit) {
++num_protected_frames_;
***plete_frame = true;
}
// Produce FEC over at most |params_.max_fec_frames| frames, or as soon as:
// (1) the excess overhead (actual overhead - requested/target overhead) is
// less than |kMaxExcessOverhead|, and
// (2) at least |min_num_media_packets_| media packets is reached.
if (***plete_frame &&
(num_protected_frames_ == params_.max_fec_frames ||
(ExcessOverheadBelowMax() && MinimumMediaPacketsReached()))) {
// We are not using Unequal Protection feature of the parity erasure code.
constexpr int kNumImportantPackets = 0;
constexpr bool kUseUnequalProtection = false;
int ret = fec_->EncodeFec(media_packets_, params_.fec_rate,
kNumImportantPackets, kUseUnequalProtection,
params_.fec_mask_type, &generated_fec_packets_);
if (generated_fec_packets_.empty()) {
ResetState();
}
return ret;
}
return 0;
}
2.1 ForwardErrorCorrection
最重要的函数是EncodeFec，大家如果对编码感兴趣可以看看前面的文章：流媒体弱网优化之路(FEC)——FEC原理简介、流媒体弱网优化之路(FEC+mediasoup)——FEC引入的问题收尾。以上两篇文章介绍了Fec的一些基础原理以及WebRTC中的UlpFec编码原理。
int ForwardErrorCorrection::EncodeFec(const PacketList& media_packets,
uint8_t protection_factor,
int num_important_packets,
bool use_unequal_protection,
FecMaskType fec_mask_type,
std::list<Packet*>* fec_packets) {
const size_t num_media_packets = media_packets.size();
// Sanity check arguments.
RTC_DCHECK_GT(num_media_packets, 0);
RTC_DCHECK_GE(num_important_packets, 0);
RTC_DCHECK_LE(num_important_packets, num_media_packets);
RTC_DCHECK(fec_packets->empty());
const size_t max_media_packets = fec_header_writer_->MaxMediaPackets();
if (num_media_packets > max_media_packets) {
RTC_LOG(LS_WARNING) << "Can't protect " << num_media_packets
<< " media packets per frame. Max is "
<< max_media_packets << ".";
return -1;
}
// Error check the media packets.
// 根据媒体队列进行非平衡编码，编码的内容则是针对payload进行。
for (const auto& media_packet : media_packets) {
RTC_DCHECK(media_packet);
if (media_packet->data.size() < kRtpHeaderSize) {
RTC_LOG(LS_WARNING) << "Media packet " << media_packet->data.size()
<< " bytes "
"is smaller than RTP header.";
return -1;
}
// Ensure the FEC packets will fit in a typical MTU.
if (media_packet->data.size() + MaxPacketOverhead() + kTransportOverhead >
IP_PACKET_SIZE) {
RTC_LOG(LS_WARNING) << "Media packet " << media_packet->data.size()
<< " bytes "
"with overhead is larger than "
<< IP_PACKET_SIZE << " bytes.";
}
}
// Prepare generated FEC packets.
// 根据保护因子和媒体数据包获取fec的包数
int num_fec_packets = NumFecPackets(num_media_packets, protection_factor);
if (num_fec_packets == 0) {
return 0;
}
for (int i = 0; i < num_fec_packets; ++i) {
generated_fec_packets_[i].data.EnsureCapacity(IP_PACKET_SIZE);
memset(generated_fec_packets_[i].data.MutableData(), 0, IP_PACKET_SIZE);
// Use this as a marker for untouched packets.
generated_fec_packets_[i].data.SetSize(0);
fec_packets->push_back(&generated_fec_packets_[i]);
}
// 创建编码表，编码表会根据fec_mask_type的类型，分两类：随机丢包 和 聚簇丢包
// 同时媒体包数量越多编码的表更大。
internal::PacketMaskTable mask_table(fec_mask_type, num_media_packets);
packet_mask_size_ = internal::PacketMaskSize(num_media_packets);
memset(packet_masks_, 0, num_fec_packets * packet_mask_size_);
// 开始根据掩码表换算重点保护包数和非平衡保护数
internal::GeneratePacketMasks(num_media_packets, num_fec_packets,
num_important_packets, use_unequal_protection,
&mask_table, packet_masks_);
// Adapt packet masks to missing media packets.
int num_mask_bits = InsertZerosInPacketMasks(media_packets, num_fec_packets);
if (num_mask_bits < 0) {
RTC_LOG(LS_INFO) << "Due to sequence number gaps, cannot protect media "
"packets with a single block of FEC packets.";
fec_packets->clear();
return -1;
}
packet_mask_size_ = internal::PacketMaskSize(num_mask_bits);
// Write FEC packets to `generated_fec_packets_`.
// 生成Fec保护Payload
GenerateFecPayloads(media_packets, num_fec_packets);
// TODO(brandtr): Generalize this when multistream protection support is
// added.
const uint32_t media_ssrc = ParseSsrc(media_packets.front()->data.data());
const uint16_t seq_num_base =
ParseSequenceNumber(media_packets.front()->data.data());
FinalizeFecHeaders(num_fec_packets, media_ssrc, seq_num_base);
return 0;
}
...
int ForwardErrorCorrection::NumFecPackets(int num_media_packets,
int protection_factor) {
// Result in Q0 with an unsigned round.
// 这个逻辑很有意思，protection_factor是8位的 最大是256, 1<<7 是128。
// 媒体包数*protection_factor + 128 之后再 >>8 就是 媒体包数*(protection_factor/255)
int num_fec_packets = (num_media_packets * protection_factor + (1 << 7)) >> 8;
// Generate at least one FEC packet if we need protection.
if (protection_factor > 0 && num_fec_packets == 0) {
num_fec_packets = 1;
}
RTC_DCHECK_LE(num_fec_packets, num_media_packets);
return num_fec_packets;
}
...
void GeneratePacketMasks(int num_media_packets,
int num_fec_packets,
int num_imp_packets,
bool use_unequal_protection,
PacketMaskTable* mask_table,
uint8_t* packet_mask) {
RTC_DCHECK_GT(num_media_packets, 0);
RTC_DCHECK_GT(num_fec_packets, 0);
RTC_DCHECK_LE(num_fec_packets, num_media_packets);
RTC_DCHECK_LE(num_imp_packets, num_media_packets);
RTC_DCHECK_GE(num_imp_packets, 0);
const int num_mask_bytes = PacketMaskSize(num_media_packets);
// Equal-protection for these cases.
if (!use_unequal_protection || num_imp_packets == 0) {
// Retrieve corresponding mask table directly:for equal-protection case.
// Mask = (k,n-k), with protection factor = (n-k)/k,
// where k = num_media_packets, n=total#packets, (n-k)=num_fec_packets.
rtc::ArrayView<const uint8_t> mask =
mask_table->LookUp(num_media_packets, num_fec_packets);
memcpy(packet_mask, &mask[0], mask.size());
} else { // UEP case
UnequalProtectionMask(num_media_packets, num_fec_packets, num_imp_packets,
num_mask_bytes, packet_mask, mask_table);
} // End of UEP modification
} // End of GetPacketMasks
...
void ForwardErrorCorrection::GenerateFecPayloads(
const PacketList& media_packets,
size_t num_fec_packets) {
RTC_DCHECK(!media_packets.empty());
for (size_t i = 0; i < num_fec_packets; ++i) {
Packet* const fec_packet = &generated_fec_packets_[i];
size_t pkt_mask_idx = i * packet_mask_size_;
const size_t min_packet_mask_size = fec_header_writer_->MinPacketMaskSize(
&packet_masks_[pkt_mask_idx], packet_mask_size_);
const size_t fec_header_size =
fec_header_writer_->FecHeaderSize(min_packet_mask_size);
size_t media_pkt_idx = 0;
auto media_packets_it = media_packets.cbegin();
uint16_t prev_seq_num =
ParseSequenceNumber((*media_packets_it)->data.data());
while (media_packets_it != media_packets.end()) {
Packet* const media_packet = media_packets_it->get();
// Should `media_packet` be protected by `fec_packet`?
if (packet_masks_[pkt_mask_idx] & (1 << (7 - media_pkt_idx))) {
size_t media_payload_length =
media_packet->data.size() - kRtpHeaderSize;
size_t fec_packet_length = fec_header_size + media_payload_length;
if (fec_packet_length > fec_packet->data.size()) {
// Recall that XORing with zero (which the FEC packets are prefilled
// with) is the identity operator, thus all prior XORs are
// still correct even though we expand the packet length here.
fec_packet->data.SetSize(fec_packet_length);
}
// 根据之前提到的编码逻辑，多个媒体包会生成一个fec包的payload，丢弃的数据需要几个媒体包一起进行恢复
XorHeaders(*media_packet, fec_packet);
XorPayloads(*media_packet, media_payload_length, fec_header_size,
fec_packet);
}
media_packets_it++;
if (media_packets_it != media_packets.end()) {
uint16_t seq_num =
ParseSequenceNumber((*media_packets_it)->data.data());
media_pkt_idx += static_cast<uint16_t>(seq_num - prev_seq_num);
prev_seq_num = seq_num;
}
pkt_mask_idx += media_pkt_idx / 8;
media_pkt_idx %= 8;
}
RTC_DCHECK_GT(fec_packet->data.size(), 0)
<< "Packet mask is wrong or poorly designed.";
}
}
2.2 码表换算
Fec码表是根据 fec_mask_type 的类型和媒体数据包数量决定使用哪个表。
const uint8_t* PacketMaskTable::PickTable(FecMaskType fec_mask_type,
int num_media_packets) {
RTC_DCHECK_GE(num_media_packets, 0);
RTC_DCHECK_LE(static_cast<size_t>(num_media_packets), kUlpfecMaxMediaPackets);
if (fec_mask_type != kFecMaskRandom &&
num_media_packets <=
static_cast<int>(fec_private_tables::kPacketMaskBurstyTbl[0])) {
return &fec_private_tables::kPacketMaskBurstyTbl[0];
}
return &fec_private_tables::kPacketMaskRandomTbl[0];
}
上述函数返回一个表的地址，在使用的时候通过下面的索引函数查找合适的Fec编码表。
rtc::ArrayView<const uint8_t> LookUpInFecTable(const uint8_t* table,
int media_packet_index,
int fec_index) {
RTC_DCHECK_LT(media_packet_index, table[0]);
// Skip over the table size.
const uint8_t* entry = &table[1];
uint8_t entry_size_increment = 2; // 0-16 are 2 byte wide, then changes to 6.
// Hop over un-interesting array entries.
for (int i = 0; i < media_packet_index; ++i) {
if (i == 16)
entry_size_increment = 6;
uint8_t count = entry[0];
++entry; // skip over the count.
for (int j = 0; j < count; ++j) {
entry += entry_size_increment * (j + 1); // skip over the data.
}
}
if (media_packet_index == 16)
entry_size_increment = 6;
RTC_DCHECK_LT(fec_index, entry[0]);
++entry; // Skip over the size.
// Find the appropriate data in the second dimension.
// Find the specific data we're looking for.
// 索引出fec的表
for (int i = 0; i < fec_index; ++i)
entry += entry_size_increment * (i + 1); // skip over the data.
size_t size = entry_size_increment * (fec_index + 1);
return {&entry[0], size};
}
当数据小于12个包时，Fec掩码表可以直接计算。
当数据大于12个包时，Fec则需要拆分出来进行掩码计算，下面就是例子。
举例：假设用13个媒体包+3个Fec包。
根据选表原则：任意一列的fec编码占8位，把每个媒体包分配到fec包里的方式是根据表的每一位是否为1来确定的。而确定一个fec保护多少个媒体包则是根据——媒体%Fec包——这个方式来确定的。下面代码中给出了一个非常长的一串位运算逻辑，这串位运算逻辑的意思是：
例如有3个fec包，那么我们需要把当前的媒体数据包都分到fec包中进行保护才能完全放进去。
通过下方的位运算之后得到一个以下的码表：
10010010
01001000
01001001
00100000
00100100
10010000
为1的位就是保护媒体数据的位一共13个。
大家发现我们一共给了3个fec包数量，但是编出来了6行掩码数据，这是为什么呢？
原因是掩码有两种:
媒体包数 < 16个时：我们是可以确认最大16个，因此使用2行来进行运算；
媒体包数 > 16个时：是一组数据，因此用6行来进行换算。
rtc::ArrayView<const uint8_t> PacketMaskTable::LookUp(int num_media_packets,
int num_fec_packets) {
RTC_DCHECK_GT(num_media_packets, 0);
RTC_DCHECK_GT(num_fec_packets, 0);
RTC_DCHECK_LE(num_media_packets, kUlpfecMaxMediaPackets);
RTC_DCHECK_LE(num_fec_packets, num_media_packets);
if (num_media_packets <= 12) {
return LookUpInFecTable(table_, num_media_packets - 1, num_fec_packets - 1);
}
int mask_length =
static_cast<int>(PacketMaskSize(static_cast<size_t>(num_media_packets)));
// Generate FEC code mask for {num_media_packets(M), num_fec_packets(N)} (use
// N FEC packets to protect M media packets) In the mask, each FEC packet
// o***upies one row, each bit / coloumn represent one media packet. E.g. Row
// A, Col/Bit B is set to 1, means FEC packet A will have protection for media
// packet B.
// Loop through each fec packet.
for (int row = 0; row < num_fec_packets; row++) {
// Loop through each fec code in a row, one code has 8 bits.
// Bit X will be set to 1 if media packet X shall be protected by current
// FEC packet. In this implementation, the protection is interleaved, thus
// media packet X will be protected by FEC packet (X % N)
for (int col = 0; col < mask_length; col++) {
fec_packet_mask_[row * mask_length + col] =
((col * 8) % num_fec_packets == row && (col * 8) < num_media_packets
? 0x80
: 0x00) |
((col * 8 + 1) % num_fec_packets == row &&
(col * 8 + 1) < num_media_packets
? 0x40
: 0x00) |
((col * 8 + 2) % num_fec_packets == row &&
(col * 8 + 2) < num_media_packets
? 0x20
: 0x00) |
((col * 8 + 3) % num_fec_packets == row &&
(col * 8 + 3) < num_media_packets
? 0x10
: 0x00) |
((col * 8 + 4) % num_fec_packets == row &&
(col * 8 + 4) < num_media_packets
? 0x08
: 0x00) |
((col * 8 + 5) % num_fec_packets == row &&
(col * 8 + 5) < num_media_packets
? 0x04
: 0x00) |
((col * 8 + 6) % num_fec_packets == row &&
(col * 8 + 6) < num_media_packets
? 0x02
: 0x00) |
((col * 8 + 7) % num_fec_packets == row &&
(col * 8 + 7) < num_media_packets
? 0x01
: 0x00);
}
}
return {&fec_packet_mask_[0],
static_cast<size_t>(num_fec_packets * mask_length)};
}
三、总结
上面补充了WebRTC查表的流程，大家可以结合前面的内容流媒体弱网优化之路(FEC+mediasoup)——FEC引入的问题收尾这篇文章提供了码表换算的一个过程，相对来说Fec里面的逻辑有很多算法的细节，使用起来还需要多多分析，下次实测一下。
