The Token Type 單詞類型

Introduction 簡介
Header 'wave/context.hpp' synopsis 頭文件概要
Template parameters 模板參數
Public Typedefs  公有typedef
Member functions 成員函數

Introduction 簡介

The token type in Wave is the main carrier of information. It is returned by dereferencing the iterators exposed by the lexing component and the iterator exposed by the preprocessor component. The tokens are originally generated by the lexer ("An entity that lexically transforms the subject of parsing to a sequence of objects (called tokens) more suitable for subsequent parsing."). The Wave library contains two different, interchangable C++ lexers, which may be used as a starting point during developing your own application. The lexer generated tokens are transformed by the preprocessing engine (macro expansion, include file processing etc.) and after this returned to the user of the Wave library.
Wave 中的 token 類型是信息的主要載體。它通過對由lexing組件暴露的迭代器或由預處理器組件暴露的迭代器進行解引用來返回得到。單詞原本是由lexer("一個實 體,將分析對像按詞彙轉換為某個對象(稱為單詞)序列以便更適合於進行下一步分析")生成的。Wave 庫帶有兩個不同的、可互換的C++ lexer,可以作為起點來開發你自己的應用程序。lexer生成的單詞將被預處理引擎轉換(宏展開、頭文件處理等等),然後返回給 Wave 庫的用戶。

You can use arbitrary token types in conjunction with your own lexer as long as these implement some required interface. The required token type interface is described below and is implemented by the wave::cpplexer::lex_token template, the required lexer interface is described here.
你可以把任意的單詞類型用於你自己的lexer,只要它們實現了所要求的接口。要求的單詞類型接口在下文描述,並通過 wave::cpplexer::lex_token 模板實現,要求的lexer接口則在這裡描述。

In the following is described the token type predefined inside the Wave library, which is used in conjunction with the two predefined C++ lexers contained in the Wave library too. If you need to use your own token type, it is required to implement the interafce described below.
以下將描述在 Wave 庫中預定義的單詞類型,它可以與兩個也是在 Wave 庫中預定義的C++ lexer共用。如果你需要使用自己的單詞類型,則必須實現下述接口。

Header wave/cpplexer/cpp_lex_token.hpp synopsis 頭文件概要

namespace boost {
namespace wave {
namespace cpplexer {

template <typename Position>
class lex_token
{
public:
typedef std::basic_string<char> string_type;
typedef Position position_type;

lex_token();
lex_token(token_id id, string_type const &value,
PositionT const &pos);

// accessors
operator token_id() const;
string_type const &get_value() const;
position_type const &get_position() const;
void set_token_id (token_id id);
void set_value (string_type const &newval);
void set_position (position_type const &pos);
};

} // namespace cpplexer } // namespace wave } // namespace boost

Template parameters 模板參數

The predefined token type uses a template parameter Position, which allows to specify the type to be used to carry the file position information contained inside the generated tokens. Your own token type do not need to take this Position template parameter, but please note, that the token type in any case needs to have an embedded type definition position_type (see below) .
預定義的單詞類型使用一個模板參數 Position,它可以指定一個類型,用於保存包含在生成的單詞中的文件位置信息。你自己的單詞類型不一定要帶有這個 Position 模板參數,不過請注意,任何情況下,單詞類型都需要具有內嵌的類型定義 position_type (見後)。

Public Typedefs 公有的typedef

The token type needs to define two embedded types: string_type and position_type. The string_type needs to be a type compatible to the std::basic_string<> class.
單詞類型需要定義兩個類型:string_typeposition_typestring_type 必須是一個兼容於 std::basic_string<> 類的類型。

This type should contain at least the filename, the line number and the column number of the position, where the token was recognized. For the predefined token type it defaults to a simple file_position template class described here. Note, that your own position_type should follow the interface described for the file_position template as well.
這個類型應至少含有文件名、識別出單詞的位置的行號和列號。對於預定義的單詞類型,缺省為一個簡單的 file_position 模板類,相關說明在此。注意,你自己的 position_type 也應遵循和 file_position 模板一樣的接口。

Member functions 成員函數

Constructors 構造函數

    lex_token();

lex_token(token_id id,
string_t const &value,
PositionT const &pos);

The first (default) constructor is for generating an end of stream token, which is used for indicating the end of the underlying input stream.
第一個(缺省)構造函數生成一個流結束單詞,用於表示底層輸入流的結束。

The second constructor initializes the newly created token object with its token id (for a list of valid token id's please look here), the string representation of its value and the file position, describing the position inside the input stream , where this token was recognized.
第二個構造函數對新創建的單詞對像以token id(有效的token id列表請見此處)、該值的字符串表示法和在輸入流中識別出該單詞的文件位置進行初始化。

Accessor functions 訪問器函數

operator token_id

    operator token_id() const;

Allows the access to the token id of the token. This accessor allows the usage of Spirit parsers directly on top top of a token stream generated by Wave. The possible token id's are described here.
允許訪問該單詞的 id。這個訪問器允許你在 Wave 生成的單詞流之上直接使用 Spirit 分析器。可用的單詞 id 在此處描述。

This function does not throw in any case.
該函數在任何情況下都不會拋出。

get_value

    string_type const &get_value() const;

Returns the value of the token, as it was recognized in the input stream. Even for constant tokens (as keywords or operators etc.) the returned value reflects the character sequence as found in the input stream.
返回該單詞的值,即在輸入流中識別到的值。即使對於常量單詞(如關鍵字或操作符等),返回值反映了在輸入流中查找到的字符序列。

This function does not throw in any case.
該函數在任何情況下都不會拋出。

get_position

    Position const &get_position() const;

Returns the position of the token in the input stream, where it was recognized. The position contains information about the filename, the line number and the column number of the token. By default the Wave library uses a file_position template for this purpose, which is described in more detail here.
返回該單詞在輸入流中被識別出來的位置。該位置含有關於文件名、單詞所在的行號及列號的信息。缺省情況下,Wave 庫使用一個 file_position 模板來表示,詳細說明請見這裡

This function does not throw in any case.
該函數在任何情況下都不會拋出。

set_token_id

    void set_token_id(token_id id);

Changes the token id of the token to the new value. The possible token id's are described here. Probably this function is of little value for the library user, but it is required under certain circumstances for correct operation of the preprocessing engine.
將該單詞的 id 改為新的值。可用的單詞 id 在此處描述。也許這個函數對於庫用戶來說用處不大,不過它對於在某些情況下預處理引擎的正常運行是必須的。

This function does not throw in any case.
該函數在任何情況下都不會拋出。

set_value

    void set_value(string_type const &newval);

Changes the value stored inside the token to the new value. Propably this function is of little value for the library user, but it is required under certain circumstances for correct operation of the preprocessing engine.
將保存在該單詞內的值改為新的值。也許這個函數對於庫用戶來說用處不大,不過它對於在某些情況下預處理引擎的正常運行是必須的。

set_position

    void set_position(Position const &newpos);

Changes the position stored inside the token to the new value. This is used for instance for implementing the functionality required for to implement the #line directive.
將保存在該單詞內的位置改為新的值。用於實現象 #line 指令所需的功能。