Help | Advanced Search
Computer Science > Computer Vision and Pattern Recognition
Title: improving gloss-free sign language translation by reducing representation density.
Abstract: Gloss-free sign language translation (SLT) aims to develop well-performing SLT systems with no requirement for the costly gloss annotations, but currently still lags behind gloss-based approaches significantly. In this paper, we identify a representation density problem that could be a bottleneck in restricting the performance of gloss-free SLT. Specifically, the representation density problem describes that the visual representations of semantically distinct sign gestures tend to be closely packed together in feature space, which makes gloss-free methods struggle with distinguishing different sign gestures and suffer from a sharp performance drop. To address the representation density problem, we introduce a simple but effective contrastive learning strategy, namely SignCL, which encourages gloss-free models to learn more discriminative feature representation in a self-supervised manner. Our experiments demonstrate that the proposed SignCL can significantly reduce the representation density and improve performance across various translation frameworks. Specifically, SignCL achieves a significant improvement in BLEU score for the Sign Language Transformer and GFSLT-VLP on the CSL-Daily dataset by 39% and 46%, respectively, without any increase of model parameters. Compared to Sign2GPT, a state-of-the-art method based on large-scale pre-trained vision and language models, SignCL achieves better performance with only 35% of its parameters. Implementation and Checkpoints are available at this https URL .
Submission history
Access paper:.
- HTML (experimental)
- Other Formats
References & Citations
- Google Scholar
- Semantic Scholar
BibTeX formatted citation
Bibliographic and Citation Tools
Code, data and media associated with this article, recommenders and search tools.
- Institution
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs .
IMAGES
VIDEO
COMMENTS
had prior courses in computer systems, some of the topics here will be familiar. If you haven't then it might not be, but don't worry. It is not rocket science, just computer science. We are going to give you a glimpse of how computers work. We will show you what puts the \digital" in digital systems and how computers process information. We
Data Representation Data Representation Eric Roberts CS 106A February 10, 2016 Claude Shannon Claude Shannon was one of the pioneers who shaped computer science in its early years. In his master's thesis, Shannon showed how it was possible to use Boolean logic and switching circuits to perform arithmetic calculations. That work led
An 8-bit code, used now only by some IBM mainframes. UNICODE. Provides a unique number for every character. A family of encodings - 8, 16, and 32 bits per character. Allows a greater variety of characters. Able to represent virtually any character in use today in any language, and some no longer in use.
Data Representation Representing Numbers • Numeric data consists of numbers that can be used in arithmetic operations. • Digital devices represent numeric data using the binary number system, also called base 2. • The binary number system only has two digits: 0 and 1. • No numeral like 2 exists in the system, so the
Chapter 4 - Data Representation The focus of this chapter is the representation of data in a digital computer. We begin with a review of several number systems (decimal, binary, octal, and hexadecimal) and a discussion of methods for conversion between the systems. The two most important methods are
1.1.2 The Binary Numbering System Most modern computer systems (including the IBM PC) operate using binary logic. The computer represents values using two voltage levels (usually 0v and +5v). With two such levels we can represent exactly two different values. These could be any two differ-ent values, but by convention we use the values zero and ...
L9.2 Data Representation The constructor for elements of recursive types is fold, while unfold destructs elements. 'e: [ˆ :˝= ]˝ 'folde: ˆ :˝ 'e: ˆ :˝ 'unfolde: [ˆ :˝= ]˝ This "unfolding" of the recursion seems like a strange operation, and it is. For example, for all other data constructors the components have a smaller
Internal representations. Usually two states, which we interpret as 0 and 1. Volatile representations: Capacitor (DRAM) charged or not. Flip-flop circuit (SRAM) one of two output signals is high. Non-volatile representations: Region of a magnetized surface (hard disks, tape) positive or negative. Floating gate transistor (flash)
CS 2401 Comp. Org. & Assembly. Data Representation in Computer 119 Systems -- Chapter 2. Representing Colors on a Video Display. An image is composed pixels (Picture elements) Different display modes use different data representations for each pixel A mixture of red, green, and blue form a specific color on the display Color depth describes ...
Abstraction is About Representation Recall our definition of abstraction from the first lecture: Abstraction is a technique used to make complex systems manageable by changing the amount of detail used to represent or interact with the system. We'll use abstraction to translate 0s and 1s to decimal numbers, then translate those numbers to other ...
unpacked BCD number has only a single decimal digit stored in each data byte. In this case, the decimal digit will be in the low four bits and the upper 4 bits of the byte will be 0. In the packed BCD representation, two decimal digits are placed in each byte. Generally, the high order bits of the data byte contain the more significant decimal ...
Data Representation Alexander Nelson September 16, 2019 ... Byte Representation Computer Architectures started to use 8-bit bytes as the basic storage size Popularized by 1970s microprocessors (Intell 8008, predecessor of ... Representation between systems should be documented to prevent miscalculation
COMP2401 - Chapter 2 - Data Representation Fall 2020 - 44 - 2.1 Number Representation and Bit Models All data stored in a computer must somehow be represented numerically in some way whether it is numerical to begin with, a series of characters or an image. Ultimately, everything digitally breaks down to ones and zeros.
• The binary, hexadecimal, and octal number systems • Finite representation of unsigned integers • Finite representation of signed integers • Finite representation of rational (floatingpoint) numbers-Why? • A power programmer must know number systems and data representation to fully understand C's . primitive data types. Primitive ...
Section 3.1 - Data Types. Registers contain either data or control information. Control information is a bit or group of bits used to specify the sequence of command signals needed for data manipulation. Data are numbers and other binary-coded information that are operated on. Possible data types in registers:
Chapter 2 - Data Representation. The focus of this chapter is the representation of data in a digital computer. We begin with a review of several number systems (decimal, binary, octal, and hexadecimal) and a discussion of methods for conversion between the systems. The two most important methods are conversion from decimal to binary and ...
When subtracting two 32-bit integers X and Y, the flags are. N = R31. Z is set if R is zero. C is clear if the result is incorrect for an unsigned subtraction. = 31& 31 ∥ 31 & 31∥ 31 & 31. V is set if the result is incorrect for an signed subtraction. = 31& 31 & 31 ∥ 31 & 31& 31.
Information representation 1 Chapter 1 1.01 Number systems Humans use the base 10 number system. Computers use digital data in the form of electrical signals. Digital data is represented as bits. Data values, such as numbers and characters, need more than a single bit. Most PCs store data as 8-bit patterns called bytes. Learning Objectives:
Commonly used to represent the mantissa of floating points (see later slides) One's Complement. Early computers and other current particular usages. Two's Complement. Most of current computers. Biased representation.
Binary Number System Digital computer represents all kinds of data and information in the binary system. Binary number system consists of two digits 0 (l ow voltage) and 1 (hi gh voltage). Its base or radix is 2. Each digit or bit in binary number system can be 0 or 1. The positional values are expressed in power of 2.
2.2 DATA REPRESENTATION IN COMPUTER A computer system is an electronic device that processes data. An electronic device, in general, consists of two stable states represented as 0 and 1. Therefore, the basic unit of data on a computer is called a Binary Digit or a Bit. With the advances in
• The binary, hexadecimal, and octal number systems • Finite representation of unsigned integers • Finite representation of signed integers ... • A power programmer must know number systems and data representation to fully understand C's primitive data types Primitive values and the operations on them. Agenda ... Computer programmers ...
A bit is a 0 or 1 used in the digital representation of data. In digital computers, the user input is first converted and transmitted as electrical pulses that can be represented by two unique states ON and OFF. The ON state may be represented by a "1" and the off state by a "0".The sequence of ON'S and OFF'S forms the electrical ...
The Platonic Representation Hypothesis. We argue that representations in AI models, particularly deep networks, are converging. First, we survey many examples of convergence in the literature: over time and across multiple domains, the ways by which different neural networks represent data are becoming more aligned.
Holistic scene understanding poses a fundamental contribution to the autonomous operation of a robotic agent in its environment. Key ingredients include a well-defined representation of the surroundings to capture its spatial structure as well as assigning semantic meaning while delineating individual objects. Classic components from the toolbox of roboticists to address these tasks are ...
Concept recommendation aims to suggest the next concept for learners to study based on their knowledge states and the human knowledge system. While knowledge states can be predicted using knowledge tracing models, previous approaches have not effectively integrated the human knowledge system into the process of designing these educational models. In the era of rapidly evolving Large Language ...
View PDF Abstract: Gloss-free sign language translation (SLT) aims to develop well-performing SLT systems with no requirement for the costly gloss annotations, but currently still lags behind gloss-based approaches significantly. In this paper, we identify a representation density problem that could be a bottleneck in restricting the performance of gloss-free SLT.